GRPO rollout server

GRPO training on my personal-style models was spending almost all of its wall clock generating completions. The trainer sat idle while the policy produced samples to score. On a single GPU, step time was 380 seconds and the model was not the bottleneck — the sampling loop was.

The fix was architectural: move rollouts to a vLLM server on a second GPU and pipeline them one step off-policy. Instead of generate-then-train-then-generate, the trainer consumes the previous step's rollouts while the server generates the next batch. Policy lag of one step is a deliberate, bounded trade — the gradient is computed on slightly stale samples in exchange for never leaving either GPU idle.

Step time went 380s → 97s → 52s, about 7× faster on 2×H200.

Three details made it work. The server had to expose the exact sampling parameters the trainer expected, including per-request temperature and top-p, because GRPO's advantage estimates are only meaningful if the samples came from the distribution the policy actually produced. Rollout metadata had to come back with the samples — token-level log-probs and completion boundaries — so the trainer never re-tokenizes text and silently changes what it thinks the policy generated. And weight synchronization had to be cheap: pushing updated adapter weights to the server each step is where naive implementations lose all the time they saved.

What I'd do differently: profile before tuning. Batch-size tuning is the most tempting knob on a rollout server and the least useful one if generation throughput is not the constraint — which is a thing to measure, not assume.

The interesting part is that this is a systems problem wearing a machine-learning costume. The objective never changed; the wall clock did.