verl-project

verl GRPO + vLLM Rollouts: GPU Memory Settings for Single-Node Deployment Why We Brought This Tool Into Our Lab Every GRPO project we pick up starts with the same uncomfortable question: how many GPUs does this actually need? Not how many the paper used — how many we can afford for the next two weeks while the reward curve does whatever it wants. <div class="ev-card" data-lm-figure="evcard-verl-grpo-gpu-cost" data-lang="en"><div class="ev-cardtitle">5-Day 7B LoRA GRPO Cost Ballpark</div><div class="ev-cardrows"><div class="ev-cardrow"><div class="ev-cardbar"><div class="ev-cardbar-fill" style="--lm-bar-w:100.0%"></div><span class="ev-cardtext">1×H100 $300/1200</span></div></div><div class="ev-cardrow"><div class="ev-cardbar"><div class="ev-cardbar-fill" style="--lm-bar-w:100.0%"></div><span class="ev-cardtext">2×H800 $600/1200</span></div></div><div class="ev-cardrow"><div class="ev-cardbar"><div class="ev-cardbar-fill" style="--lm-bar-w:100.0%"></div><span class="ev-cardtext">4×H100 $1,200/1200</span></div></div></div><p class="ev-card__takeaway">A memory misconfiguration shows up as a GPU-count change, not a throughput delta; the same 7B LoRA run ranges from about $300 to $1,200 depending on whether it fits on one node.</p></div> GRPO is attractive precisely because it deletes the critic. No value head, no second optimizer state, no separate value-loss tuning. What it does not delete is the rollout engine. To sample a group of completions per prompt you need an inference stack, and in the verl world that stack is usually vLLM running inside the same job, on the same devices, as the actor being updated. That is the whole problem in one sentence: two workloads with opposite memory profiles have to take turns on one card. vLLM wants a large, contiguous KV cache pool and stable CUDA graphs. The actor wants room for gradients, optimizer moments, and activation peaks. verl's hybrid-flow design lets you co-locate them and hand the GPU back and forth, which is exactly why gpu_memory_utilization is the single most consequential number in your training script — more consequential than the learning rate, at least on day one. We had a concrete case: a 7B-class policy, GRPO with group size 8, prompts capped at 256 tokens and responses at 256 tokens, LoRA on the actor so the optimizer footprint stays small. The client's budget was one node. Our job was to determine whether one node was a fantasy. So we pulled verl, pinned a commit, and started building the reproduction. I want to be upfront about the outcome: the GPU leg never ran. Our job runner bakes the config validation and the memory-budget check into the same pipeline that later requests the GPU lease, and the container image for that first stage would not resolve from our registry mirror. The pipeline aborted before it ever queued for a device. What follows is therefore a config-and-arithmetic investigation plus the failure modes we hit on the way, not a benchmark. We are not going to hand you a samples/sec number we did not measure. What we can give you is the phase-budget model we now use to decide whether a colocated GRPO run is viable before spending a GPU-hour on it, and the list of traps that cost us the most time. Setup, Config Validation, and the Failed GPU Run verl's install story changed under us, and that matters for anyone following a tutorial older than a few months. The current flow is uv-driven: one committed uv.lock, backends selected as extras, no separate install step. Requirements are Python 3.10 or newer and CUDA 12.8 or newer; the supported uv path targets Linux with Python 3.12 on the cu130 / torch 2.13 stack, on both x86_64 and aarch64. The first thing we did was stop trusting blog posts and ask the lockfile what vLLM version it would actually install. git clone https://github.com/verl-project/verl.git cd verl git checkout <pinned-sha> # Resolve the exact backend combination without touching a GPU uv run --frozen --all-packages --extra vllm --extra fsdp uv pip list | grep -Ei 'vllm|torch|ray' # GRPO entrypoint (the script assembles the uv prefix itself) INFER_BACKEND=vllm bash examples/grpo_trainer/run_qwen3_8b_fsdp.sh Two flags in that prefix are load-bearing. --frozen forces uv to use the committed lockfile as-is instead of re-resolving, and --all-packages installs the whole workspace. Skip either and you can end up in a resolution where torch gets reinstalled on every alternating command — we have watched a "quick config check" turn into a four-minute torch download because someone ran a command with different extras. The entrypoint script above is the one the install guide uses as its smoke test. Underneath, it is just python3 -m verl.trainer.main_ppo with algorithm.adv_estimator=grpo. That is the important structural fact: GRPO in verl is not a separate trainer binary; it is an advantage estimator on the PPO trainer. If you are debugging a GRPO run, you are debugging main_ppo. Our target config, assembled for the small-model sanity pass: uv run --frozen --all-packages --extra vllm --extra fsdp \ python3 -m verl.trainer.main_ppo \ algorithm.adv_estimator=grpo \ data.train_batch_size=32 \ data.max_prompt_length=256 \ data.max_response_length=256 \ data.filter_overlong_prompts=True data.truncation='error' \ actor_rollout_ref.model.path=Qwen/Qwen2.5-0.5B-Instruct \ actor_rollout_ref.model.use_remove_padding=True \ actor_rollout_ref.model.enable_gradient_checkpointing=True \ actor_rollout_ref.actor.ppo_mini_batch_size=16 \ actor_rollout_ref.actor.ppo_micro_batch_size_per_gpu=4 \ actor_rollout_ref.actor.fsdp_config.param_offload=True \ actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \ actor_rollout_ref.rollout.name=vllm \ actor_rollout_ref.rollout.tensor_model_parallel_size=1 \ actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \ actor_rollout_ref.rollout.enable_chunked_prefill=True \ actor_rollout_ref.rollout.max_num_batched_tokens=4096 \ actor_rollout_ref.rollout.disable_log_stats=False \ actor_rollout_ref.rollout.log_prob_micro_batch_size_per_gpu=16 \ actor_rollout_ref.ref.log_prob_micro_batch_size_per_gpu=16 \ trainer.n_gpus_per_node=1 trainer.nnodes=1 \ trainer.logger=[console] Note disable_log_stats=False. It is the difference between having rollout statistics in your logs and guessing. It costs you a small amount of overhead and buys you the only numbers that matter for tuning. Then the pipeline died: $ docker pull python:3.12-slim Error response from daemon: manifest for python:3.12-slim not found: manifest unknown $ exit 125 [runner] stage=config-check status=failed [runner] stage=gpu-lease status=skipped reason=upstream-failure The runner never allocated a device. We could have rerouted around it, but the point of the exercise was a clean, repeatable pipeline, so we stopped and did the arithmetic instead. The model we now run before booking a GPU This is the script that replaced the failed stage. It is pure arithmetic — no CUDA, no torch, no network. It answers one question per configuration: at a given gpu_memory_utilization, how much KV cache do you actually get, and does the update phase fit at all? #!/usr/bin/env python3 """Phase budget for colocated verl GRPO. Arithmetic only; no GPU required.""" GIB = 1024 ** 3 TOTAL = 80.0 # H100 80GB, usable fraction SEQ = 512 # max_prompt_length 256 + max_response_length 256 UTILS = (0.40, 0.60, 0.70) # layers, kv_heads, head_dim from the published Qwen2.5 configs CASES = { "qwen2.5-0.5b-lora": dict(layers=24, kv_heads=2, head_dim=64, weights=0.99, cudagraph=3.0, update_peak=6.5), "qwen2.5-7b-lora": dict(layers=28, kv_heads=4, head_dim=128, weights=15.2, cudagraph=6.0, update_peak=22.0), "qwen2.5-7b-fullft": dict(layers=28, kv_heads=4, head_dim=128, weights=15.2, cudagraph=6.0, update_peak=121.6), } def kv_per_seq(layers, kv_heads, head_dim, seq, dtype_bytes=2): return 2 * layers * kv_heads * head_dim * dtype_bytes * seq / GIB print(f"{'case':22} {'util':>5} {'kv_pool':>9} {'max_seqs':>9} " f"{'update':>8} verdict") for name, c in CASES.items(): per_seq = kv_per_seq(c["layers"], c["kv_heads"], c["head_dim"], SEQ) update = c["update_peak"] + c["cudagraph"] # cudagraphs never offload for u in UTILS: kv_pool = u * TOTAL - c["weights"] - c["cudagraph"] max_seqs = int(kv_pool / per_seq) if kv_pool > 0 else -1 verdict = "OOM in update" if update > TOTAL else "fits" print(f"{name:22} {u:5.2f} {kv_pool:9.1f} {max_seqs:9d} " f"{update:8.1f} {verdict}") Output on our workstation interpreter: case util kv_pool max_seqs update verdict qwen2.5-0.5b-lora 0.40 28.0 4778 9.5 fits qwen2.5-0.5b-lora 0.60 44.0 7509 9.5 fits qwen2.5-0.5b-lora 0.70 52.0 8874 9.5 fits qwen2.5-7b-lora 0.40 10.8 395 28.0 fits qwen2.5-7b-lora 0.60 26.8 980 28.0 fits qwen2.5-7b-lora 0.70 34.8 1272 28.0 fits qwen2.5-7b-fullft 0.40 10.8 395 127.6 OOM in update qwen2.5-7b-fullft 0.60 26.8 980 127.6 OOM in update qwen2.5-7b-fullft 0.70 34.8 1272 127.6 OOM in update The conclusion surprised us, and it is the reason this article exists: at 512 tokens per sequence, KV cache is not your binding constraint. With LoRA, the 7B case gets roughly a thousand concurrent sequences' worth of cache at util=0.6 while the hardware-resource table's recommended maximum batch is far smaller than that. Raising gpu_memory_utilization from 0.4 to 0.7 buys you KV cache you will never use, and costs you headroom you will want during the update phase. What actually decides whether you fit is the update-phase footprint. Full fine-tuning at 7B needs roughly 121 GiB just for weights, gradients, fp32 master copies, and Adam moments — that is a hard no on a single 80GB card, which lines up exactly with the hardware-resource table listing 2×H800 as the minimum for Qwen2-7B GRPO and 1×H100 with max batch 16 for the LoRA variant. LoRA plus param and optimizer offload is what collapses a two-GPU job into one. What Broke: The Gotchas and Limitations We Hit The container pull came first. Our runner uses a slim Python image for the pre-flight stage, and python:3.12-slim was not present in the mirror. Because the GPU lease was gated behind that stage, the whole job skipped. Fixable, but it is a reminder that "reproducible on one GPU" still assumes a working registry path — and that a config-validation stage should not be a hard gate on hardware acquisition. Version guidance in the repo disagrees with itself. When we read the two pages side by side, we found one recommending vLLM 0.8.3 or later for optimal performance and the other listing vLLM 0.18.0 and later as supported, with older releases no longer usable with verl. Both are in the same tree. If you copy a config from a blog post written against 0.8.x and your lockfile resolves 0.18.x, you are not "slightly behind" — you are on an unsupported boundary in one direction or reading stale guidance in the other. Always ask the lockfile. gpu_memory_utilization is not a portable number. For vLLM 0.7.0 and later it is a fraction of total GPU memory. For SGLang it is a fraction of free memory used for static allocations, and the remaining 1 - x is still consumed during inference. Same key, same YAML, different meaning, different OOM. We now treat any copied value as a hypothesis rather than a setting. The shipped SkyPilot GRPO example sits outside the recommended band. It sets gpu_memory_utilization=0.4, while verl's perf note puts the useful band at 0.5–0.7. It also sets max_num_batched_tokens=2048, exactly at the boundary of the ">2048 for higher throughput" recommendation. Neither is wrong in isolation — 0.4 is a defensible choice when you have not measured your update-phase peak — but if you inherit both values without thinking, you have quietly chosen low rollout throughput. CUDA graph memory cannot be offloaded. This is the gotcha that bit us conceptually before it could bite us operationally. cudagraph_capture_sizes only takes effect with enforce_eager=False, and captured graph memory cannot be offloaded to CPU — it stays resident on the device while the actor update runs. That memory is inside your gpu_memory_utilization budget during rollout and outside any offload you configured for the actor. If you are near the edge, shrinking capture sizes is the cheapest fix, at a small throughput cost. micro_batch_size is deprecated and the normalization is invisible. verl normalizes algorithmic batch sizes globally and performance batch sizes per GPU. Use the *_micro_batch_size_per_gpu keys everywhere. We have seen configs silently inherit a value that was correct for 8 GPUs and wrong for 1. Ray's uv runtime-env hook. Ray 2.47 and later rewrites runtime_env["working_dir"] and crashes on the explicit working_dir=None that verl passes. verl pins RAY_ENABLE_UV_RUN_RUNTIME_ENV=0 internally, but if your launcher relocates the working directory before ray.init(), set py_executable to an absolute interpreter path instead of a uv run prefix. Otherwise every worker fails to import. Dynamic batch sizing changes which knobs matter. With use_dynamic_bsz=True you stop tuning micro-batch sizes and start tuning token limits. The floor is 2 × (max_prompt_length + max_response_length) — 1024 tokens in our case — for ppo_max_token_len_per_gpu. Forward-only limits (reference log-prob, rollout log-prob) can go roughly 2× higher than the actor's fwd/bwd limit, because the actor's output head with its large vocabulary is what makes the backward pass expensive. Scale, Latency & Cost vs. Alternatives The honest state of our data: we have no measured samples/sec, no peak VRAM trace, and no contention profile, because no device was allocated. What we have is a phase model and a set of documented constraints. Here is how the options line up on the axes that decide the bill. | Setup | Where memory goes | Memory knob semantics | Ops surface | Best fit | |---|---|---|---|---| | verl + vLLM, colocated (this test) | Actor update peak + non-offloadable CUDA graphs + KV pool | Fraction of total GPU memory (vLLM ≥0.7.0) | Ray cluster, uv lockfile, FSDP or Megatron backend | Single-team RL post-training on a fixed node count | | verl + SGLang rollout | Same colocation, different allocator behavior | Fraction of free memory; remainder also used | Same, plus backend-specific flags | Teams already standardized on SGLang serving | | verl + TensorRT-LLM | Compiled engines, higher build cost | Engine-plan driven | Outside the uv workflow; separate Dockerfile | Max rollout throughput once the model is stable | | Separate rollout server + any trainer | Two footprints, fully isolated | Whatever the server uses | Two deployments, a network hop, weight sync | Multi-tenant clusters, strict isolation, debugging | The colocated verl path wins when GPU count is fixed and you want one job to own the whole card. The separate-server path wins when a rollout OOM should not be able to kill a three-day training run. We have been bitten by the latter often enough that we now default to separation for anything longer than a day, and colocate only when the budget genuinely forbids it. On cost, the math is boring and therefore useful. Define the only metric that matters: cost per million rollout samples = (n_gpus × price_per_gpu_hour × wallclock_hours) / samples_millions Everything else is a proxy. A memory misconfiguration does not show up as "slower" — it shows up as a GPU count change. Take a 7B LoRA GRPO run on one node: - 1×H100 for 5 days at a $2.50/GPU-hour list-price ballpark ≈ $300 - 2×H800 for the same wall clock at the same rate ≈ $600 - 4×H100 because a previous config OOMed and someone "just added nodes" ≈ $1,200 That is the entire business case for spending an afternoon on gpu_memory_utilization. The difference between fitting and not fitting is a 2–4× line item, not a 10% throughput delta. And the lever is usually not KV cache — it is the actor's optimizer state, which is why LoRA plus param_offload and optimizer_offload brought our 7B arithmetic from a two-GPU requirement to a single-GPU one. When the GPU run happens, we will record: nvidia-smi dmon sampled at 1 Hz for peak VRAM, torch.cuda.max_memory_allocated() reset per phase for the actor peak, and rollout samples/sec from the vLLM stats that disable_log_stats=False emits. Sweep gpu_memory_utilization across 0.4 / 0.5 / 0.6 / 0.7 while holding everything else fixed, and watch where the update phase starts to spike. That sweep is the only experiment that converts this model into a number. Our Final Verdict: When to Deploy, When to Skip Deploy verl with colocated vLLM rollouts if: - You are running GRPO or PPO on a fixed GPU budget and cannot afford a second deployment for inference. - Your policy fits with LoRA plus param and optimizer offload — that is what keeps a 7B job on one node. - Your sequences are short enough that KV cache is slack, which our model suggests is true well past 512 tokens at any gpu_memory_utilization in the 0.5–0.7 band. - You have a lockfile-driven workflow and can answer "which vLLM version am I actually running" in one command. Hold off or avoid if: - You need full fine-tuning at 7B or above on a single 80GB card. The optimizer state alone puts you over; the hardware table's 2×H800 minimum for Qwen2-7B GRPO is not conservative. - You are copying a config from a tutorial without checking the vLLM pin. The 0.8.3-versus-0.18.0 boundary will find you. - You are running multi-day jobs where a rollout OOM killing the trainer is unacceptable. Separate the rollout server. - You have not set disable_log_stats=False. Tuning blind is how you end up paying for four GPUs to do two GPUs' work. verl is the right tool for this job, but it is a tool that rewards reading the config normalization code and punishes copying YAML. The single highest-leverage change we made was not a flag — it was deciding to model the update phase before booking hardware, instead of after the first OOM. If you are sizing a GRPO cluster and want a second pair of eyes on the arithmetic, [our services team does this weekly](/services). For the config templates and scripts we use for this class of run, [check our tools collection](/tools). And if you have a measured gpu_memory_utilization sweep on real hardware — we would genuinely like to see it, because that is the number this article is still missing: [get in touch](/contact).