Learning to Act While Waiting

RL Finetuning of Generalist Robot Policies Under Inference Latency

Brian Zhu*¹, Momen Khalil*¹, E. H. Harrison*², Emanuele Poggi*¹, Philipp Sebastian Schmitt¹, Bernd Kast¹, Philine Meister¹, Pranav Atreya², Qiyang Li², Finn Ferchau¹, Cesar Colmenero¹, Yash Shahapurkar¹, Gokul Narayanan¹, Melih Erdogan¹, Kai M. Wurm¹, Georg von Wichert¹, Oier Mees^3,4,2, Eugen Solowjow¹, Andrew Wagenmaker², Sergey Levine²

¹ Siemens ² UC Berkeley ³ Microsoft ⁴ ETH Zurich

* Equal contribution

Paper

Abstract

While reinforcement learning (RL) allows generalist robot policies to continually improve during deployment, the large model size of modern generalist policies, such as VLAs, poses a fundamental obstacle to effective RL improvement. In particular, their severe inference latency—which can lead to pauses or jerky movements—can alter the effective environment dynamics and, if not correctly accounted for, break the Markov assumption that RL relies on, causing standard RL algorithms to fail completely. In this work, we introduce a latency-aware framework, Asynchronous RL with Intermediate Information (ARLI), that enables RL-based improvement of generalist policies under inference delays. Our framework builds on asynchronous inference approaches, which interleave action generation with execution to hide latency, and addresses its incompatibility with RL by providing a low-latency RL policy design that maximizes reactivity within the inference window through two contributions: state augmentations that restore near-Markovian structure by incorporating committed actions and a mid-inference observation. We evaluate our approach across simulated and real-world manipulation tasks, and find that it enables effective finetuning under inference delays where standard RL fails entirely, even matching or exceeding the performance of standard RL in idealized no-latency settings.

Asynchronous RL with Intermediate Information (ARLI)

Enabling RL improvement of generalist robot policies under inference latency. Generalist policies incur significant action inference latency delays, either forcing pauses during inference or relying on stale observations, making RL improvement brittle. ARLI uses intermediate actions and updated asynchronous observations to steer the next action chunk, enabling effective real-world improvement under asynchronous inference.

Large vision-language-action models (VLAs) can require 100–300+ ms to generate actions on consumer GPUs. Such delays can cause pauses or jerky motions that alter the effective environment dynamics and degrade performance, especially in settings that require high reactivity. Although asynchronous inference strategies compute the next action while the current action is being executed to mitigate these effects, they introduce a key challenge for RL: since inference begins before the current action finishes executing, the policy cannot condition on the actual state where the action will be played. Naively combining RL with asynchronous inference produces a non-Markovian effective state, causing standard RL methods to fail.

ARLI overcomes this non-Markovian state with careful state augmentation and a low-latency RL policy design. We build on Diffusion Steering via Reinforcement Learning (DSRL), which trains a lightweight RL policy to select the input noise that steers a generalist flow or diffusion policy toward desired behaviors. Under asynchronous inference, naive DSRL conditioning on only the delayed initial state is insufficient for effective finetuning.

An overview of asynchronous inference and RL finetuning under inference delays. Use the arrows or dots to compare naive asynchronous inference, DSRL baselines, and ARLI.

Our two core contributions restore near-Markovian structure within the inference window:

Intermediate actions: Include committed actions executed since inference began, giving the RL policy information about how the environment evolves before the next chunk plays.
Intermediate observation: Condition on a more recent state at t − t_rl^delay, after the VLM backbone finishes but before the action expert completes denoising, maximizing reactivity without adding inference delay.

The benefit of intermediate state conditioning under disturbances.

The benefit of intermediate state conditioning. In naive asynchronous inference, the initial state does not capture stochastic events (e.g., moving obstacles). By conditioning on an intermediate state, the policy can react to disturbances that occur after inference begins.

We further show that when environments exhibit open-loop consistency, delayed state information does not incur significant suboptimality, providing theoretical support for ARLI’s state augmentations. Real-Time Chunking (RTC) can also be combined with ARLI to improve temporal consistency during finetuning.

Simulated Experiments

We evaluate ARLI on high-reactivity simulated tasks, including Kinetix environments (mjc swimmer, mjc walker, car launch) and AlohaTransferCube, comparing against naive asynchronous DSRL and DSRL with RTC.

Comparison of ARLI and DSRL on simulation tasks with and without RTC.

ARLI significantly outperforms naive DSRL under asynchronous inference, converging to higher final success rates with less training time. While DSRL with RTC improves finetuning on some tasks, it is insufficient on its own; the state augmentations in ARLI are required for reliable improvement. In many cases ARLI performs effectively without RTC, though RTC further improves efficiency and reliability.

Real-World Experiments

We test whether ARLI enables effective RL finetuning on a bimanual UR5e robot cell with wrist and base cameras. All experiments initialize from π_0.5 finetuned on small human demonstration datasets at 60 Hz with action chunk length 50.

Comparison of ARLI against other RL finetuning methods on real-world tasks. Each task shows successful and unsuccessful rollout videos (top) and training curves with estimated throughput (bottom).

We consider three real-world manipulation tasks:

Assembly: Insert a power connector into an inverter box by aligning it with two pins.
Shoe-in-Bag: Pick up a shoe and place it into a narrow bag.
Bag-Placement: With both arms, pick up a lying bag and position it parallel to the front.

ARLI substantially outperforms synchronous DSRL, DSRL with RTC, and other asynchronous baselines, achieving higher success rates and greater estimated throughput across all three tasks.

Real-World Rollout Videos

ARLI finetuning rollouts on real-world manipulation tasks.

Ablations

We ablate the components of ARLI, including intermediate actions, intermediate observations, RTC, and varying inference delays. Additional figures from the paper are shown below.

State augmentation ablations

Variants of ARLI with RTC ablations. — Comparison of ARLI and DSRL with RTC ablations.

ARLI performance at varying RL inference delays.

Delay sensitivity and noise analysis

Delay sensitivity on mjc swimmer with and without RTC. — Delay sensitivity on *mjc swimmer* with and without RTC.

Checkpoint-wise evolution of the ARLI noise distribution. — Checkpoint-wise evolution of the ARLI noise distribution on held-out tasks.

Video Presentation

BibTeX

@misc{zhu2026learningactwaitingrl,
      title={Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latency}, 
      author={Brian Zhu and Momen Khalil and E Harrison and Emanuele Poggi and Philipp Schmitt and Bernd Kast and Philine Meister and Pranav Atreya and Qiyang Li and Finn Ferchau and Cesar Colmenero and Yash Shahapurkar and Gokul Narayanan and Melih Erdogan and Kai Wurm and Georg von Wichert and Oier Mees and Eugen Solowjow and Andrew Wagenmaker and Sergey Levine},
      year={2026},
      eprint={2608.23831},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2608.23831}
    }