ai-dynamo

Optimizing GPU Resource Planning with NVIDIA Dynamo GPU resource allocation for LLM inference presents complex optimization challenges. Workloads exhibit variable request patterns, unpredictable sequence lengths, and strict latency requirements, making static GPU allocation inefficient. Overprovisioning wastes expensive compute resources, while underprovisioning violates service level agreements (SLAs) and degrades user experience. NVIDIA Dynamo's SLA Planner automates GPU resource management through SLA-driven profiling and intelligent autoscaling, analyzing performance requirements (time-to-first-token, inter-token latency), profiling workload characteristics, and dynamically adjusting GPU worker replicas to meet latency targets while minimizing resource consumption. This guide covers deploying Dynamo's SLA Planner on Kubernetes, including infrastructure setup with Prometheus monitoring, DynamoGraphDeploymentRequest (DGDR) configuration for automated profiling and deployment, and Grafana dashboard integration for visualizing scaling decisions and performance metrics. By the end, you'll have a Kubernetes-based LLM deployment that automatically scales GPU workers to meet SLA targets. --- Prerequisites Before you begin, ensure you: - Have access to a Kubernetes cluster with NVIDIA GPUs configured with the NVIDIA GPU Operator. Use a cluster with at least 2 GPU nodes. - Install kubectl and configure it to access your cluster. - Install Helm 3 for package management. - Install jq for JSON processing in command-line operations. - Create an NVIDIA NGC account and generate an NGC API key with NGC Catalog access for pulling container images. - Create a Hugging Face account and generate an access token for gated models like Llama. - Have cluster-admin permissions to install Custom Resource Definitions (CRDs) and create namespaces. Key Components This deployment uses several components that work together to provide automated GPU resource planning. - NVIDIA Dynamo Platform provides the orchestration layer with the Dynamo Operator that manages DynamoGraphDeployment (DGD) and DynamoGraphDeploymentRequest (DGDR) resources. The operator automates the profiling, configuration generation, and deployment lifecycle. - SLA Planner acts as the autoscaling controller that monitors inference metrics from Prometheus and adjusts worker replica counts to meet SLA targets. It uses load predictors (constant, ARIMA, Kalman, or Prophet) to forecast request rates and sequence lengths, then calculates optimal GPU allocations based on TTFT (time-to-first-token) and ITL (inter-token latency) requirements. - DynamoGraphDeploymentRequest (DGDR) serves as the declarative interface for deploying models with performance constraints. Users specify the model name, backend type (vLLM, SGLang, or TensorRT-LLM), SLA targets, and container images. The operator automatically profiles the configuration and generates an optimized DynamoGraphDeployment. - DynamoGraphDeployment (DGD) defines the low-level implementation details for a deployed model, including frontend services, prefill workers, decode workers, and the SLA planner component. - Prometheus collects inference metrics from the Dynamo frontend, including request counts, TTFT, ITL, input sequence length (ISL), and output sequence length (OSL). - Grafana visualizes planner decisions, worker counts, GPU usage, observed metrics versus predicted metrics, and correction factors. 1. Verify GPU Availability 1. List GPU nodes in your cluster: $ kubectl get nodes -o json | jq '.items[] | select(.status.capacity."nvidia.com/gpu") | {name: .metadata.name, gpus: .status.capacity."nvidia.com/gpu"}' The output displays nodes with available GPUs. { "name": "gpu-node-1", "gpus": "2" } { "name": "gpu-node-2", "gpus": "2" } 2. Verify GPU resources are allocatable: $ kubectl describe nodes | grep -A 5 "nvidia.com/gpu" Ensure at least one node shows nvidia.com/gpu: 1 or higher under the Allocatable section. 2. Label GPU Nodes 1. Check if GPU nodes are already labeled: $ kubectl get nodes --show-labels | grep nvidia.com/gpu.present If the output displays nodes with the nvidia.com/gpu.present=true label, skip to the next section. If no output appears, proceed with labeling. 2. Identify your GPU nodes: $ kubectl get nodes 3. Label each GPU node with the Dynamo GPU label. Replace NODE_NAME with your GPU node name: $ kubectl label node NODE_NAME nvidia.com/gpu.present=true Repeat this command for each GPU node in your cluster. 4. Verify the labels were applied: $ kubectl get nodes --show-labels | grep nvidia.com/gpu.present 3. Run Pre-Deployment Checks 1. Clone the Dynamo repository if you haven't already: $ git clone https://github.com/ai-dynamo/dynamo.git $ cd dynamo 2. Check out the latest stable release. Visit the Dynamo releases page to find the latest version: $ git checkout release/0.9.0 3. Run the pre-deployment check script: $ ./deploy/pre-deployment/pre-deployment-check.sh The script validates kubectl connectivity, default StorageClass configuration, GPU resources, and GPU Operator installation. ======================================== Dynamo Pre-Deployment Check Script ======================================== ✅ kubectl Connectivity: PASSED ✅ Default StorageClass: PASSED ✅ Cluster GPU Resources: PASSED ✅ GPU Operator: PASSED All pre-deployment checks passed! Your cluster is ready for Dynamo deployment. If any checks fail, review the error messages and resolve the issues before proceeding. 4. Install Prometheus 1. Add the Prometheus Helm repository: $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts $ helm repo update 2. Install the kube-prometheus-stack with configuration for Dynamo metric scraping: $ helm install prometheus -n monitoring --create-namespace prometheus-community/kube-prometheus-stack \ --set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false \ --set-json 'prometheus.prometheusSpec.podMonitorNamespaceSelector={}' \ --set prometheus.prometheusSpec.probeNamespaceSelector.matchLabels=null 3. Verify Prometheus is running: $ kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus 5. Install Dynamo Platform 1. Set environment variables for the installation: $ export NAMESPACE=dynamo-system $ export RELEASE_VERSION=0.9.0 2. Navigate to the home directory: $ cd ~ 3. Fetch the Dynamo CRDs Helm chart: $ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-crds-${RELEASE_VERSION}.tgz 4. Install the CRDs: $ helm install dynamo-crds dynamo-crds-${RELEASE_VERSION}.tgz 5. Fetch the Dynamo Platform Helm chart: $ helm fetch https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${RELEASE_VERSION}.tgz 6. Install the Dynamo Platform with the operator enabled and the Prometheus endpoint configured: $ helm upgrade --install dynamo-platform dynamo-platform-${RELEASE_VERSION}.tgz \ --namespace ${NAMESPACE} \ --create-namespace \ --set dynamo-operator.controllerManager.manager.image.tag=0.9.0 \ --set grove.enabled=true \ --set kai-scheduler.enabled=true \ --set nats.reloader.image.tag=0.22.3 \ --set prometheusEndpoint=http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090 The command configures the following components: - --namespace ${NAMESPACE} and --create-namespace: Installs the platform in the dynamo-system namespace, creating it if it doesn't exist. - --set dynamo-operator.controllerManager.manager.image.tag=0.9.0: Specifies the Dynamo Operator version that manages DGD and DGDR lifecycle operations. - --set grove.enabled=true: Enables Grove, the distributed runtime that coordinates communication between frontend and worker services. - --set kai-scheduler.enabled=true: Enables the KAI (Kubernetes AI) scheduler for advanced GPU scheduling and resource allocation. - --set nats.reloader.image.tag=0.22.3: Specifies the NATS configuration reloader version that automatically reloads NATS server configuration changes without downtime. - --set prometheusEndpoint=...: Configures the SLA Planner to query inference metrics from the Prometheus service in the monitoring namespace. > If the installation fails with a Grove webhook certificate conflict error, uninstall the release and reinstall: > > ```console > $ helm uninstall dynamo-platform --namespace ${NAMESPACE} > ``` > > Then run the installation command again. The warning "tls: failed to find any PEM data in certificate input" is harmless and can be ignored. 7. Verify the operator is running: $ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/name=dynamo-operator NAME READY STATUS RESTARTS AGE dynamo-platform-dynamo-operator-controller-manager-77464cd6pcjt 2/2 Running 0 45s 6. Create Image Pull Secrets 1. Export your NGC API key. Replace YOUR_NGC_API_KEY with your NGC API key from the NGC portal: $ export NGC_API_KEY=YOUR_NGC_API_KEY 2. Export your Hugging Face token. Replace YOUR_HF_TOKEN with your Hugging Face access token: $ export HF_TOKEN=YOUR_HF_TOKEN 3. Create a Docker registry secret for NGC access using the exported variable: $ kubectl create secret docker-registry nvcr-imagepullsecret \ --docker-server=nvcr.io \ --docker-username='$oauthtoken' \ --docker-password=${NGC_API_KEY} \ -n ${NAMESPACE} 4. Create a Hugging Face token secret for model downloads using the exported variable: $ kubectl create secret generic hf-token-secret \ --from-literal=HF_TOKEN=${HF_TOKEN} \ -n ${NAMESPACE} 7. Select AI Configurator System and Backend Version The AI Configurator performs offline profiling to generate optimal deployment configurations without running actual workloads. The aicSystem parameter specifies your GPU hardware type, and the aicBackendVersion parameter specifies the inference backend version for profiling data. Available GPU Systems | System | Description | Supported Backends | |--------|-------------|-------------------| | a100_sxm | NVIDIA A100 SXM | TensorRT-LLM | | b200_sxm | NVIDIA B200 SXM | SGLang, TensorRT-LLM | | gb200_sxm | NVIDIA GB200 SXM | TensorRT-LLM | | h100_sxm | NVIDIA H100 SXM | vLLM, SGLang, TensorRT-LLM | | h200_sxm | NVIDIA H200 SXM | vLLM, SGLang, TensorRT-LLM | | l40s | NVIDIA L40S | SGLang, TensorRT-LLM | Backend Version Compatibility vLLM Backend | GPU System | Supported Versions | |------------|-------------------| | H100 SXM | 0.12.0 | | H200 SXM | 0.12.0 | Example configuration for vLLM: sweep: useAiConfigurator: true aicSystem: h200_sxm aicBackendVersion: "0.12.0" SGLang Backend | GPU System | Supported Versions | |------------|-------------------| | B200 SXM | 0.5.6.post2 | | H100 SXM | 0.5.6.post2 | | H200 SXM | 0.5.6.post2 | | L40S | 0.5.5.post3 | Example configuration for SGLang: sweep: useAiConfigurator: true aicSystem: h200_sxm aicBackendVersion: "0.5.6.post2" TensorRT-LLM Backend | GPU System | Supported Versions | |------------|-------------------| | A100 SXM | 1.0.0 | | B200 SXM | 1.0.0rc6, 1.2.0rc5 | | GB200 SXM | 1.0.0rc6, 1.2.0rc5 | | H100 SXM | 1.0.0rc3, 1.2.0rc5 | | H200 SXM | 1.0.0rc3, 1.2.0rc5 | | L40S | 1.0.0 | Example configuration for TensorRT-LLM: sweep: useAiConfigurator: true aicSystem: h200_sxm aicBackendVersion: "1.2.0rc5" Selecting the Correct Configuration Match your configuration to your cluster's GPU hardware: identify your GPU type with kubectl describe nodes, select the matching aicSystem code, and choose a compatible backend version from the tables above. An unsupported combination causes the profiling job to fail with an error indicating no profiling data is available for that configuration. 8. Deploy a Model with SLA-Driven Profiling 1. Create a DGDR manifest file with SLA specifications for the NVIDIA Nemotron Nano 4B model using the vLLM backend: $ cat << 'EOF' > nemotron-nano-4b-dgdr.yaml apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeploymentRequest metadata: name: nemotron-nano-sla spec: model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 backend: vllm autoApply: false profilingConfig: profilerImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 config: sla: isl: 2000 osl: 100 ttft: 300.0 itl: 30.0 sweep: useAiConfigurator: true aicSystem: h200_sxm aicHfId: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 aicBackendVersion: "0.12.0" planner: plannerMinEndpoint: 1 plannerMaxGpuBudget: 4 plannerAdjustmentInterval: 30 deploymentOverrides: workersImage: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 name: vllm-disagg EOF The manifest specifies a Nemotron Nano 4B model with SLA targets (ISL 2000, OSL 100, TTFT 300ms, ITL 30ms), AI Configurator offline profiling for H200 SXM GPUs (20-30 seconds versus 2-4 hours for online profiling), a planner budget of 1-4 GPUs with 30-second adjustment intervals, and autoApply: false so you can review the generated DGD before deployment. > AI Configurator System Selection: The aicSystem parameter must match your GPU hardware type. Available options are: a100_sxm, b200_sxm, gb200_sxm, h100_sxm, h200_sxm, and l40s. The aicBackendVersion parameter must match the available version for your selected system and backend combination. For vLLM backend: H100/H200 SXM supports version 0.12.0. For SGLang: H100/H200 SXM and B200 SXM support version 0.5.6.post2, L40S supports 0.5.5.post3. For TensorRT-LLM: consult the AI Configurator documentation for supported versions. 2. Apply the DGDR manifest: $ kubectl apply -f nemotron-nano-4b-dgdr.yaml -n ${NAMESPACE} 3. Monitor the DGDR status: $ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -w The DGDR transitions through states: Pending → Profiling → Ready. NAME MODEL BACKEND STATE DGD-STATE AGE nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Pending 5s nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Profiling 10s nemotron-nano-sla nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 vllm Ready 35s 4. View detailed DGDR status and events: $ kubectl describe dgdr nemotron-nano-sla -n ${NAMESPACE} The output displays profiling results and the generated DGD configuration in the status.generatedDeployment field. 9. Extract and Deploy the Generated DGD 1. Extract the generated DGD configuration to a file: $ kubectl get dgdr nemotron-nano-sla -n ${NAMESPACE} -o jsonpath='{.status.generatedDeployment}' > nemotron-nano-dgd-raw.yaml 2. Add required Pod metadata environment variables and the --trust-remote-code flag to worker services using jq: $ cat nemotron-nano-dgd-raw.yaml | jq '.spec.services |= with_entries( .value.extraPodSpec.mainContainer.env = [ {"name":"POD_UID","valueFrom":{"fieldRef":{"fieldPath":"metadata.uid"}}}, {"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}}, {"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}, {"name":"POD_IP","valueFrom":{"fieldRef":{"fieldPath":"status.podIP"}}} ] | if .value.componentType == "worker" then if (.value.extraPodSpec.mainContainer.args // []) | any(. == "--trust-remote-code") | not then .value.extraPodSpec.mainContainer.args += ["--trust-remote-code"] else . end else . end )' > nemotron-nano-dgd.yaml This injects Pod metadata environment variables into all services (for Grove distributed runtime coordination) and adds the --trust-remote-code flag only to worker service arguments. 3. Apply the modified DGD configuration to deploy the model: $ kubectl apply -f nemotron-nano-dgd.yaml -n ${NAMESPACE} 4. Verify the DynamoGraphDeployment was created: $ kubectl get dgd -n ${NAMESPACE} NAME AGE vllm-disagg 30s 5. Check that the frontend and worker pods are running: $ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/part-of=vllm-disagg Wait until all pods show Running status before proceeding. Initial deployment may take several minutes as worker pods download the model and load it into GPU memory; monitor with kubectl get pods -n ${NAMESPACE} -w. 10. Test the Deployment 1. Get the frontend service name: $ kubectl get svc -n ${NAMESPACE} -l nvidia.com/dynamo-discovery-backend=kubernetes | grep frontend vllm-disagg-frontend ClusterIP 10.43.69.246 <none> 8000/TCP 2m 2. Forward the frontend service to your local machine: $ kubectl port-forward -n ${NAMESPACE} svc/vllm-disagg-frontend 8000:8000 Keep this terminal session running. 3. In a new terminal, send a test inference request: $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": [ { "role": "user", "content": "Explain GPU resource planning in one sentence." } ], "max_tokens": 50, "temperature": 0.7 }' 4. Press Ctrl+C in the port-forward terminal to stop forwarding. > Production Deployments: Port-forwarding is suitable for local testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose the frontend service with HTTPS ingress, load balancing, and certificate management. 11. Install Grafana Dashboard 1. Apply the Grafana dashboard ConfigMap: $ kubectl apply -n monitoring -f https://raw.githubusercontent.com/ai-dynamo/dynamo/release/0.9.0/deploy/observability/k8s/grafana-planner-dashboard-configmap.yaml 2. Expose the Grafana service using a NodePort for quick testing: $ kubectl patch svc prometheus-grafana \ -n monitoring \ -p '{"spec": {"type": "NodePort"}}' > NodePort is for testing only. For production deployments with secure external access, configure Gateway API with TLS encryption to expose Grafana with HTTPS ingress and certificate management. 3. Get the NodePort assigned to Grafana: $ kubectl get svc prometheus-grafana -n monitoring -o jsonpath='{.spec.ports[0].nodePort}'; echo 4. Retrieve the Grafana admin password: $ kubectl get secret prometheus-grafana -n monitoring -o jsonpath='{.data.admin-password}' | base64 -d; echo 5. Access Grafana in your browser. Replace NODE_IP with your cluster node's IP address and NODE_PORT with the port from the previous step: http://NODE_IP:NODE_PORT Log in with username admin and the password retrieved in the previous step, then navigate to Dashboards → Dynamo Planner Dashboard to view planner metrics. 12. Monitor Automatic Scaling The SLA Planner continuously monitors request rates, input/output sequence lengths, TTFT, and ITL metrics from Prometheus, then adjusts prefill and decode worker replicas every adjustment interval (30 seconds in this configuration) to meet SLA targets. Understanding Planner Behavior As your application receives inference requests, the planner: 1. Collects Metrics: Gathers request rates, sequence lengths, and latency metrics from the Dynamo frontend via Prometheus. 2. Predicts Load: Uses the configured load predictor (constant, ARIMA, Kalman, or Prophet) to forecast future traffic patterns. 3. Calculates Requirements: Determines optimal worker counts needed to meet TTFT and ITL SLA targets based on predicted load. 4. Scales Workers: Increases or decreases prefill and decode worker replicas by updating the DynamoGraphDeployment. 5. Applies Corrections: Adjusts predictions based on observed performance versus expected performance using correction factors. The dashboard visualizes worker counts, cumulative GPU-hours consumed, observed request/TTFT/ITL metrics, load predictions, correction factors, and a timeline of scaling decisions. Use the Namespace dropdown to filter metrics for your specific deployment namespace (dynamo-system in this guide). Monitor planner logs: $ kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=vllm-disagg-0-planner,app.kubernetes.io/part-of=vllm-disagg --tail=50 -f 2026-02-28T00:01:53.952354Z INFO planner_core.run: New adjustment interval started! 2026-02-28T00:01:53.962025Z INFO planner_core.observe_metrics: Observed num_req: 0.00 isl: 0.00 osl: 0.00 2026-02-28T00:01:53.962037Z INFO planner_core.observe_metrics: Observed ttft: 0.00ms itl: 0.00ms 2026-02-28T00:01:53.962463Z INFO planner_core.predict_load: Predicted load: num_req=0.00, isl=0.00, osl=0.00 2026-02-28T00:01:53.962592Z INFO planner_core._compute_replica_requirements: Predicted number of engine replicas: prefill=1, decode=1 2026-02-28T00:01:53.966200Z INFO kubernetes_connector.set_component_replicas: prefill component VllmPrefillWorker already at desired replica count 1, skipping Press Ctrl+C to stop following the logs. 13. Understanding Load Predictors The SLA Planner uses load predictors to forecast future request patterns and proactively scale resources. Choose from four prediction models based on your workload's traffic patterns. Constant — assumes the next interval's load equals the current load; use for stable workloads with long adjustment intervals. planner: loadPredictor: constant ARIMA — uses auto-ARIMA to fit optimal model parameters for time-series data with trends and seasonality. planner: loadPredictor: arima loadPredictorLog1p: true Kalman — provides fast online forecasting with smooth adaptation using a local linear trend Kalman filter. planner: loadPredictor: kalman kalmanQLevel: 0.1 kalmanQTrend: 0.01 kalmanR: 1.0 Prophet — handles complex seasonal patterns and trend changes using Facebook's Prophet model. planner: loadPredictor: prophet prophetWindowSize: 1000 loadPredictorLog1p: true Next Steps - Tune plannerMaxGpuBudget and plannerAdjustmentInterval for your real traffic patterns instead of the fast demo values used here. - Swap in ARIMA, Kalman, or Prophet load predictors once you have enough traffic history to evaluate forecasting accuracy. - Put Gateway API with TLS in front of the frontend and Grafana services before exposing either outside your cluster. - Try a different backend (SGLang or TensorRT-LLM) and GPU system combination from the AI Configurator tables to compare profiling results. For the full guide with additional tips, visit the original article on Vultr Docs.

3rd merged PR in NVIDIA Dynamo 🎉 every DCGM scrape was leaking a socket. Now the operator reuses one HTTP client instead of a fresh transport per scrape. dynamo#17387 t.co

Building Disaggregated Inference with NVIDIA Dynamo NVIDIA Dynamo is an open-source inference framework for deploying large-scale generative AI models across multi-node, multi-GPU environments. Disaggregated serving architecture separates prompt processing (prefill) and token generation (decode) phases across different GPU workers, enabling independent scaling and optimization of each phase based on their distinct computational characteristics. This guide covers building disaggregated inference deployments with NVIDIA Dynamo using AIConfigurator for optimal configuration discovery, deploying on Kubernetes with RDMA, and validating performance. Disaggregated architecture can achieve up to 1.7x throughput improvement over aggregated serving for large models (32B+) with long input contexts; for smaller models, AIConfigurator automatically determines when aggregated serving is more efficient. By the end, you'll have used AIConfigurator to find an optimal serving configuration, deployed it on a GPU Kubernetes cluster with RDMA-accelerated KV cache transfer, and validated throughput with AIPerf. --- Prerequisites Before you begin, ensure you have one of the following environments: For Kubernetes Deployment: - A Kubernetes cluster with NVIDIA GPU nodes and the RDMA device plugin installed. - kubectl configured to access your cluster. - An RDMA-capable network (InfiniBand or RoCE). - The NVIDIA Dynamo platform installed on your cluster, including the Dynamo Operator and GPU resource management configured. For Standalone Instance Deployment: - A GPU-enabled server with NVIDIA GPUs and the NVIDIA Container Toolkit configured. - 2+ GPUs for disaggregated serving. - Docker Engine and Docker Compose. - A non-root user with sudo privileges. Common Requirements: - Python 3.10+ for running AIConfigurator. - A Hugging Face account with an access token for gated models. Understanding Disaggregated Serving Architecture Disaggregated serving splits LLM inference into two specialized phases, each optimized for different computational characteristics. Prefill Phase vs Decode Phase | Phase | Characteristics | Optimization Focus | |-------|----------------|-------------------| | Prefill | Processes entire input prompt in parallel | Memory-bound, benefits from high-bandwidth GPUs, uses low batch sizes (1-4) | | Decode | Generates tokens sequentially, one at a time | Compute-bound, benefits from high concurrency, uses high batch sizes (256-1024) | Architecture Components Prefill Workers: - Process incoming prompts and generate KV cache. - Run with low batch sizes for optimal TTFT. - Can use different tensor parallelism (TP) than decode workers. - Transfer KV cache to decode workers via NIXL. Decode Workers: - Receive KV cache from prefill workers. - Generate output tokens sequentially. - Handle high concurrency with large batch sizes. - Optimized for sustained token throughput. Request Routing: - Frontend routes requests to available prefill workers. - PrefillRouter selects prefill workers based on KV cache affinity. - After prefill, decode workers receive KV cache and continue generation. - NIXL enables GPU-to-GPU KV cache transfer with RDMA. When to Use Disaggregated Serving Use Disaggregated Serving When: - Model size is large (32B+ parameters) — smaller models have lower KV cache transfer overhead that doesn't justify phase separation. - Input sequence length (ISL) significantly exceeds output sequence length (OSL) — typically ISL > 4000 tokens. - Workload requires independent scaling of prefill and decode capacity. - RDMA-capable network infrastructure is available (critical requirement). - Maximizing per-GPU throughput is the primary objective. Use Aggregated Serving When: - Model size is small (<10B parameters) — KV cache transfer overhead outweighs benefits. - Balanced ISL/OSL ratios (between 2:1 and 10:1). - Simpler deployment with lower operational complexity is preferred. - RDMA infrastructure is not available. - Strict latency SLAs require minimal inter-worker communication overhead. Performance Benefits (for Large Models): - Up to 1.7x higher throughput compared to aggregated serving for large models with long-context workloads. - Better GPU utilization through phase-specific optimization. - Independent scaling of prefill and decode resources based on workload patterns. > Note: RDMA is required. Disaggregated serving requires RDMA for KV cache transfer. Without RDMA, performance degrades by 40x (TTFT increases from ~400ms to 10+ seconds) as the system falls back to TCP transfer, creating a severe bottleneck. 1. Install and Configure AIConfigurator AIConfigurator is a performance optimization tool that automatically determines the optimal configuration for deploying LLMs with Dynamo by evaluating both aggregated and disaggregated architectures. 1. Create a Python 3 virtual environment: $ python3 -m venv aiconfigurator_env 2. Activate the virtual environment: $ source aiconfigurator_env/bin/activate 3. Install AIConfigurator using pip: $ pip3 install aiconfigurator 4. Verify the installation: $ aiconfigurator version The output displays the installed version. 5. View available parameters and help documentation: $ aiconfigurator cli default --help Understanding AIConfigurator Parameters AIConfigurator requires the following key parameters: | Parameter | Description | Example | |-----------|-------------|---------| | --hf_id | HuggingFace model ID or local path | nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 | | --system | GPU system type | h200_sxm, h100_sxm, a100_sxm | | --total_gpus | Number of GPUs available for deployment | 4 | | --isl / --osl | Input/Output sequence lengths in tokens | 4000 / 500 | | --ttft / --tpot | SLA targets - Time To First Token (ms) and Time Per Output Token (ms) | 600 / 16.67 | | --backend | Inference backend | vllm, trtllm, sglang | | --backend_version | Backend version | 0.12.0 (for vLLM) | | --save_dir | Directory to save generated configs | ./results_vllm | How AIConfigurator Works: 1. Evaluates all possible configurations (tensor/pipeline parallelism combinations). 1. Simulates performance for both aggregated and disaggregated architectures. 1. Filters configurations that meet SLA targets (TTFT, TPOT). 1. Ranks results by per-GPU throughput (tokens/s/gpu). 1. Automatically selects the best architecture (agg or disagg) based on actual performance. 1. Generates deployment manifests for top-ranked configurations. AIConfigurator considers factors like model size, KV cache transfer overhead, and GPU utilization to determine whether disaggregated serving will outperform aggregated for your specific workload. 2. Find the Optimal Disaggregated Configuration Use AIConfigurator to discover the best configuration for your model and hardware. 1. Run AIConfigurator with your model and system specifications: $ aiconfigurator cli default \ --hf_id nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 \ --system h200_sxm \ --total_gpus 4 \ --isl 4000 \ --osl 500 \ --ttft 600 \ --tpot 16.67 \ --backend vllm \ --backend_version 0.12.0 \ --save_dir ./results_vllm The command evaluates all possible configurations for 4x H200 GPUs with the specified model and SLA targets, generating deployment manifests in the ./results_vllm directory. 2. Review the AIConfigurator output summary: ******************************************************************************** * Dynamo aiconfigurator Final Results * ******************************************************************************** ---------------------------------------------------------------------------- Input Configuration & SLA Target: Model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 (is_moe: False) Total GPUs: 4 Best Experiment Chosen: agg at 3785.20 tokens/s/gpu (disagg 0.81x better) ---------------------------------------------------------------------------- Overall Best Configuration: - Best Throughput: 15,140.82 tokens/s - Per-GPU Throughput: 3785.20 tokens/s/gpu - Per-User Throughput: 61.41 tokens/s/user - TTFT: 395.88ms - TPOT: 16.28ms - Request Latency: 8521.26ms ---------------------------------------------------------------------------- The summary shows that for this 4B model, aggregated serving performs better than disaggregated (3785.20 vs 3067.51 tokens/s/gpu). This is expected for smaller models where the overhead of KV cache transfer outweighs the benefits of phase separation. Disaggregated serving becomes advantageous with larger models (32B+) and longer context lengths. 3. Review the detailed configuration comparison: agg Top Configurations: (Sorted by tokens/s/gpu) +------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+ | Rank | tokens/s/gpu | tokens/s/user | TTFT | request_latency | concurrency | total_gpus (used) | replicas | parallel | bs | +------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+ | 1 | 3785.20 | 61.41 | 395.88 | 8521.26 | 256 (=64x4) | 4 (4=4x1) | 4 | tp1pp1 | 64 | | 2 | 3036.90 | 65.06 | 331.10 | 8000.51 | 192 (=96x2) | 4 (4=2x2) | 2 | tp2pp1 | 96 | | 3 | 2305.76 | 65.36 | 249.55 | 7883.93 | 144 (=144x1) | 4 (4=1x4) | 1 | tp4pp1 | 144| +------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+ disagg Top Configurations: (Sorted by tokens/s/gpu) +------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+ | Rank | tokens/s/gpu | tokens/s/user | TTFT | request_latency | concurrency | total_gpus (used) | replicas | (p)parallel | (d)parallel | +------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+ | 1 | 3067.51 | 63.99 | 111.33 | 7909.20 | 208 (=104x2) | 4 (4=2x2) | 2 | tp1pp1 | tp1pp1 | | 2 | 3011.39 | 62.82 | 111.33 | 8054.41 | 208 (=208x1) | 4 (4=1x4) | 1 | tp1pp1 | tp2pp1 | +------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+ Understanding the Output: - tokens/s/gpu: Overall throughput efficiency — higher is better (agg: 3785.20 vs disagg: 3067.51 = agg is 23% faster). - tokens/s/user: Per-request generation speed (inverse of TPOT). - TTFT: Predicted time to first token (395.88ms for agg, 111.33ms for disagg prefill). - concurrency: Total concurrent requests (e.g., 256 (=64x4) = batch size 64 x 4 replicas). - parallel: Tensor/pipeline parallelism (tp=tensor parallel, pp=pipeline parallel). - Result: For this 4B model, aggregated serving is more efficient due to lower overhead. > Note: model size matters. Disaggregated serving becomes advantageous with larger models (32B+) where the benefits of phase-specific optimization outweigh KV cache transfer overhead. For smaller models like this 4B example, aggregated serving is more efficient. 4. Examine the generated directory structure: $ tree results_vllm/ results_vllm/ └── nvidia/ └── Llama-3.1-Nemotron-Nano-4B-v1.1_isl4000_osl500_ttft600_tpot16_472853/ ├── agg/ │ ├── best_config_topn.csv │ ├── config.yaml │ ├── pareto.csv │ ├── top1/ │ │ ├── generator_config.yaml │ │ ├── k8s_deploy.yaml │ │ └── run_0.sh │ ├── top2/ │ │ └── ... │ └── top3/ │ └── ... ├── disagg/ │ ├── best_config_topn.csv │ ├── config.yaml │ ├── pareto.csv │ ├── top1/ │ │ ├── generator_config.yaml │ │ ├── k8s_deploy.yaml │ │ └── run_0.sh │ └── top2/ │ └── ... └── pareto_frontier.png The results are organized by model name and experiment parameters. While aggregated serving is optimal for this 4B model, the following sections demonstrate disaggregated deployment for learning purposes and larger model scenarios. 3. Deploy on Kubernetes > Note: while AIConfigurator determined that aggregated serving is optimal for this 4B model, the following sections demonstrate disaggregated deployment for learning purposes. Use disaggregated architecture when deploying larger models (32B+) or when AIConfigurator recommends it for your specific workload. Kubernetes deployment provides the infrastructure for disaggregated serving with RDMA-accelerated KV cache transfer. Install the NVIDIA Dynamo Platform Before deploying disaggregated workloads, install the NVIDIA Dynamo platform (the Dynamo Operator) on your Kubernetes cluster, configure GPU resource management, set up the DynamoGraphDeployment CRD, and verify the cluster is ready for Dynamo workloads. Prepare the Kubernetes Environment 1. Set your Kubernetes namespace as an environment variable. Replace YOUR_NAMESPACE with the namespace where you want to deploy Dynamo workloads: $ export LLM_NAMESPACE=YOUR_NAMESPACE 2. Create the namespace: $ kubectl create namespace $LLM_NAMESPACE 3. Export your Hugging Face token. Replace YOUR_HF_TOKEN with your Hugging Face access token: $ export HF_TOKEN=YOUR_HF_TOKEN 4. Create a Hugging Face token secret for accessing gated models: $ kubectl create secret generic hf-token-secret \ --namespace $LLM_NAMESPACE \ --from-literal=HF_TOKEN="$HF_TOKEN" 5. Create a persistent volume claim for caching model weights: $ cat << EOF > model-cache-pvc.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: model-cache namespace: $LLM_NAMESPACE spec: accessModes: - ReadWriteMany resources: requests: storage: 100Gi EOF > Note: this PVC uses ReadWriteMany (RWX) access mode, which allows multiple pods to mount the volume simultaneously for shared model caching. Ensure your cluster has a StorageClass that supports RWX access mode (such as NFS, CephFS, or cloud-provider RWX storage). Verify available StorageClasses: > > ```console > $ kubectl get storageclass > ``` 6. Apply the PVC configuration: $ kubectl apply -f model-cache-pvc.yaml 7. Verify the RDMA device plugin is running: $ kubectl get pods -n kube-system | grep rdma The output should show RDMA device plugin pods running on GPU nodes. If not installed, follow the Mellanox RDMA device plugin installation guide. 8. Verify RDMA resources are available on your nodes. Replace NODE-NAME with your GPU node name: $ kubectl get node NODE-NAME -o json | jq '.status.allocatable' The output should show rdma/hca_shared_devices resources: { ...... "rdma/hca_shared_devices": "1k", ...... } The rdma/hca_shared_devices: "1k" entry confirms RDMA resources are available. If this resource is missing, verify the RDMA device plugin DaemonSet configuration and node labels. Deploy Using Generated Manifests 1. Review the generated disaggregated deployment manifest: $ cat results_vllm/nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1_isl4000_osl500_ttft600_tpot16_*/disagg/top1/k8s_deploy.yaml The file contains a DynamoGraphDeployment custom resource with frontend, prefill workers, and decode workers configured according to AIConfigurator's recommendations. 2. Create the complete deployment manifest with RDMA configuration: $ cat << EOF > dynamo-disagg-deployment.yaml apiVersion: nvidia.com/v1alpha1 kind: DynamoGraphDeployment metadata: name: dynamo-disagg namespace: $LLM_NAMESPACE spec: backendFramework: vllm pvcs: - name: model-cache create: false services: Frontend: componentType: frontend replicas: 1 volumeMounts: - name: model-cache mountPoint: /opt/models envs: - name: HF_HOME value: /opt/models extraPodSpec: mainContainer: image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 imagePullPolicy: IfNotPresent VLLMPrefillWorker: envFromSecret: hf-token-secret componentType: worker subComponentType: prefill replicas: 2 resources: limits: gpu: "1" sharedMemory: size: 16Gi volumeMounts: - name: model-cache mountPoint: /opt/models envs: - name: HF_HOME value: /opt/models - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid - name: UCX_TLS value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc" - name: UCX_RNDV_SCHEME value: "get_zcopy" - name: UCX_RNDV_THRESH value: "0" extraPodSpec: mainContainer: image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 workingDir: /workspace imagePullPolicy: IfNotPresent securityContext: capabilities: add: ["IPC_LOCK"] resources: limits: rdma/hca_shared_devices: "1" requests: rdma/hca_shared_devices: "1" command: ["python3", "-m", "dynamo.vllm"] args: - --model - "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1" - "--tensor-parallel-size" - "1" - "--kv-cache-dtype" - "fp8" - "--max-num-seqs" - "1" - --trust-remote-code - --is-prefill-worker VLLMDecodeWorker: envFromSecret: hf-token-secret componentType: worker subComponentType: decode replicas: 2 resources: limits: gpu: "1" sharedMemory: size: 16Gi volumeMounts: - name: model-cache mountPoint: /opt/models envs: - name: HF_HOME value: /opt/models - name: POD_UID valueFrom: fieldRef: fieldPath: metadata.uid - name: UCX_TLS value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc" - name: UCX_RNDV_SCHEME value: "get_zcopy" - name: UCX_RNDV_THRESH value: "0" extraPodSpec: mainContainer: image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 workingDir: /workspace imagePullPolicy: IfNotPresent securityContext: capabilities: add: ["IPC_LOCK"] resources: limits: rdma/hca_shared_devices: "1" requests: rdma/hca_shared_devices: "1" command: ["python3", "-m", "dynamo.vllm"] args: - --model - "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1" - "--tensor-parallel-size" - "1" - "--kv-cache-dtype" - "fp8" - "--max-num-seqs" - "1024" - --trust-remote-code - --is-decode-worker EOF Critical RDMA Configuration: - rdma/hca_shared_devices resources: Request RDMA resources matching tensor parallelism size (1 for both prefill and decode with TP1). - IPC_LOCK capability: Required for RDMA memory registration. - POD_UID environment variable: Retrieves pod UID from metadata for worker identification. - UCX_TLS environment variable: Enables RDMA transports (rc_x, dc_x for InfiniBand). - UCX_RNDV_SCHEME=get_zcopy: Enables zero-copy RDMA transfers for maximum performance. - UCX_RNDV_THRESH=0: Forces RDMA usage for all message sizes. - --trust-remote-code flag: Allows execution of custom model code from Hugging Face. - CUDA 13 runtime: Uses nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 for GPU compatibility. 3. Deploy the configuration to Kubernetes: $ kubectl apply -f dynamo-disagg-deployment.yaml 4. Monitor the deployment progress: $ kubectl get pods -n $LLM_NAMESPACE -w Wait for all pods to reach Running status. This may take several minutes as models are downloaded and loaded. Verify the Kubernetes Deployment 1. Check pod status and resource allocation: $ kubectl get pods -n $LLM_NAMESPACE The output shows all pods running successfully: NAME READY STATUS RESTARTS AGE dynamo-disagg-0-frontend-77bn5 1/1 Running 0 2m42s dynamo-disagg-0-vllmdecodeworker-gzgbw 1/1 Running 0 2m42s dynamo-disagg-0-vllmdecodeworker-kn8xz 1/1 Running 0 2m42s dynamo-disagg-0-vllmprefillworker-2b7hm 1/1 Running 0 2m42s dynamo-disagg-0-vllmprefillworker-7p4mq 1/1 Running 0 2m42s 2. Verify RDMA is active by checking worker logs. Replace PREFILL-WORKER-POD with your prefill worker pod name from the previous step: $ kubectl logs -n $LLM_NAMESPACE PREFILL-WORKER-POD | grep -i "UCX\|NIXL" The output shows NIXL initialization logs: 2026-03-03T22:52:23.152441Z INFO nixl_connector: NIXL is available 2026-03-03T22:52:23.154049Z INFO factory.create_connector: Creating v1 connector with name: NixlConnector and engine_id: e0dc9d5a-80b5-4b06-aeb9-05958f83d6ea (Worker pid=1241) 2026-03-03 22:52:25 NIXL INFO _api.py:363 Backend UCX was instantiated (Worker pid=1241) 2026-03-03 22:52:25 NIXL INFO _api.py:253 Initialized NIXL agent: 93612abb-847f-435f-bd56-73bfc8d5f3d6 2026-03-03T22:52:25.487423Z INFO nixl_connector.register_kv_caches: Registering KV_Caches. use_mla: False, kv_buffer_device: cuda, use_host_buffer: False The key indicators confirm RDMA is working: - "NIXL is available": NIXL connector initialized successfully. - "Backend UCX was instantiated": UCX backend (RDMA transport) is active. - "Initialized NIXL agent": NIXL agent ready for KV cache transfer. - "Registering KV_Caches": KV cache buffers registered for GPU-to-GPU transfer. 3. Test inference through the frontend service: $ kubectl port-forward -n $LLM_NAMESPACE service/dynamo-disagg-frontend 8000:8000 In a separate terminal, send a test request: $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": [{"role": "user", "content": "Explain the importance of tensor parallelism in LLM inference."}], "max_tokens": 100 }' The output displays the model's response in JSON format. 4. Monitor worker metrics to verify disaggregated flow: $ curl -s localhost:8000/metrics | grep dynamo_frontend_requests_total The output displays the metrics showing successful request processing: # HELP dynamo_frontend_requests_total Total number of LLM requests processed # TYPE dynamo_frontend_requests_total counter dynamo_frontend_requests_total{endpoint="chat_completions",model="nvidia/llama-3.1-nemotron-nano-4b-v1.1",request_type="unary",status="success"} 1 4. Deploy on Standalone Instances For standalone instance deployments (non-Kubernetes), refer to the backend-specific deployment guides that cover disaggregated serving: - Deploy vLLM Disaggregated Serving: 2 prefill workers + 2 decode workers with NIXL-based KV cache transfer. - Deploy TensorRT-LLM Disaggregated Serving: TRT-LLM disaggregated configuration with KV cache optimization. - Deploy SGLang Disaggregated Serving: SGLang disaggregated setup with bootstrap_info transfer. These guides provide complete deployment scripts with environment variable configuration, container runtime setup with GPU allocation, NIXL configuration for inter-worker KV cache transfer, and verification/troubleshooting steps. 5. Validate Performance with AIPerf After deployment, validate AIConfigurator's predictions against actual performance using AIPerf. 1. Install the AIPerf benchmarking tool: $ pip3 install aiperf 2. Derive AIPerf parameters from the AIConfigurator output. Values used for testing: - --concurrency 30 (AIConfigurator recommends 208 for max throughput) - --num-requests 800 (Recommended: 8320 = concurrency x 40 for statistical stability) - --isl 4000 --osl 500 (Match your AIConfigurator inputs) - --extra-inputs "ignore_eos:true" (Ensures exact OSL tokens generated) 3. Run the AIPerf benchmark against the deployment: $ aiperf profile \ -m nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 \ --endpoint-type chat \ -u http://localhost:8000 \ --isl 4000 --isl-stddev 0 \ --osl 500 --osl-stddev 0 \ --num-requests 800 \ --concurrency 30 \ --streaming \ --extra-inputs "ignore_eos:true" \ --num-warmup-requests 40 \ --ui-type simple > Note: the command above uses --concurrency 30 and --num-requests 800 for early success and faster testing. For best throughput validation, use the recommended values: --concurrency 208 and --num-requests 8320 (concurrency x 40 for statistical stability). The benchmark runs 800 requests with 30 concurrent connections. The output displays performance metrics: NVIDIA AIPerf | LLM Metrics ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓ ┃ Metric ┃ avg ┃ min ┃ max ┃ p99 ┃ p90 ┃ p50 ┃ std ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩ │ Time to First Token (ms) │ 496.77 │ 55.53 │ 8,411.38 │ 6,972.51 │ 64.59 │ 59.78 │ 1,606.66 │ │ Time to Second Token (ms) │ 0.24 │ 0.00 │ 4.72 │ 3.55 │ 0.00 │ 0.00 │ 0.81 │ │ Time to First Output Token (ms) │ 496.77 │ 55.53 │ 8,411.38 │ 6,972.51 │ 64.59 │ 59.78 │ 1,606.66 │ │ Request Latency (ms) │ 2,367.45 │ 1,505.01 │ 9,807.65 │ 8,869.04 │ 1,996.11 │ 1,972.56 │ 1,528.95 │ │ Inter Token Latency (ms) │ 3.75 │ 2.74 │ 3.90 │ 3.89 │ 3.85 │ 3.83 │ 0.25 │ │ Output Token Throughput Per User │ 268.03 │ 256.55 │ 365.49 │ 357.48 │ 279.88 │ 260.84 │ 22.84 │ │ (tokens/sec/user) │ │ │ │ │ │ │ │ │ Output Sequence Length (tokens) │ 499.53 │ 487.00 │ 510.00 │ 502.00 │ 500.10 │ 500.00 │ 1.30 │ │ Input Sequence Length (tokens) │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 0.00 │ │ Output Token Throughput (tokens/sec) │ 6,232.31 │ N/A │ N/A │ N/A │ N/A │ N/A │ N/A │ │ Request Throughput (requests/sec) │ 12.48 │ N/A │ N/A │ N/A │ N/A │ N/A │ N/A │ │ Request Count (requests) │ 800.00 │ N/A │ N/A │ N/A │ N/A │ N/A │ N/A │ └──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘ Benchmark Duration: 64.12 sec 4. Compare actual vs predicted performance: | Metric | AIConfigurator Prediction (2 GPUs) | AIPerf Actual (2 GPUs) | Status | |--------|--------------------------|---------------|--------| | TTFT (ms) | 111.33 | 59.78 (p50) / 496.77 (avg) | p50 close to prediction | | ITL/TPOT (ms) | 15.62 (1000/63.99) | 3.75 | 4x better than predicted | | Throughput (tokens/s) | 6,135.02 (3,067.51 x 2) | 6,232.31 | 101.6% - exceeds prediction | | Per-GPU Throughput | 3,067.51 tokens/s/gpu | 3,116 tokens/s/gpu | Matches prediction | The per-GPU throughput (6,232.31 / 2 GPUs = 3,116 tokens/s/gpu) closely matches AIConfigurator's prediction of 3,067.51 tokens/s/gpu. The total throughput (6,232.31 tokens/s) actually exceeds the prediction for 2 GPUs (6,135.02 tokens/s). This benchmark was run with 2 GPUs (1 prefill + 1 decode worker); scaling to 4 GPUs (2 prefill + 2 decode workers as shown in the deployment manifest) would double the throughput to approximately 12,464 tokens/s. The lower throughput compared to maximum capacity is due to using concurrency 30 instead of the recommended 208. The Inter Token Latency (3.75ms) significantly outperforms predictions, demonstrating efficient token generation. > Note: run benchmarks inside the Kubernetes cluster (using a Job pod) to eliminate network latency. External benchmarking adds latency that affects TTFT measurements but provides realistic end-to-end performance metrics. Configuration Best Practices GPU Allocation Strategies Prefill Workers: - Allocate fewer GPUs with higher memory bandwidth (memory-bound workload). - Use lower tensor parallelism (TP1 or TP2) to minimize communication overhead. - Deploy multiple replicas for higher prefill throughput. Decode Workers: - Allocate more GPUs for parallel token generation (compute-bound workload). - Use higher tensor parallelism (TP4 or TP8) to distribute computation. - Single replica with high batch size often performs best. Example Configuration (4 GPUs): - 2x Prefill workers with TP1 (2 GPUs total, 1 GPU each). - 2x Decode workers with TP1 (2 GPUs total, 1 GPU each). - Achieves balanced prefill and decode capacity with minimal tensor parallelism overhead. Tensor Parallelism Considerations | Worker Type | Recommended TP | Reasoning | |-------------|---------------|-----------| | Prefill | TP1 or TP2 | Minimizes all-reduce communication overhead, prefill is memory-bound | | Decode | TP4 or TP8 | Distributes compute load, decode is compute-bound | Batch Size Tuning Prefill Workers: - Use low batch size (1-4) for optimal TTFT. - Higher batch sizes increase memory pressure and TTFT latency. - Prefill throughput comes from multiple workers, not high batch size. Decode Workers: - Use high batch size (256-1024) for optimal throughput. - Decode operations are compute-bound and benefit from high concurrency. - Adjust based on available GPU memory and model KV cache requirements. RDMA Requirements Network Requirements: - InfiniBand (preferred) or RoCE network with RDMA support. - Sufficient bandwidth: 200 Gbps+ recommended for optimal performance. - Low latency: <10μs for RDMA operations. Verification: - Check worker logs for "NIXL is available" and "Backend UCX was instantiated" messages. - Monitor RDMA device utilization: rdma resource show. - Measure RDMA bandwidth: ib_write_bw or rdma_bw tools. Next Steps - Re-run AIConfigurator against a 32B+ model to see disaggregated serving's throughput advantage materialize. - Scale prefill and decode replicas independently based on observed ISL/OSL ratios in production traffic. - Run AIPerf with the full recommended concurrency (208) and request count (8320) for a production-grade validation. - Review the backend-specific deployment guides (vLLM, TensorRT-LLM, SGLang) if you need a standalone, non-Kubernetes setup. For the full guide with additional tips, visit the original article on Vultr Docs.

Super cool to see the community interest in DeepInfer support for Dynamo! dynamo#16985 dynamo#17008 We have a PR with disagg support we hope to drop tomorrow, so if you'd like to collaborate, join us on slack! ai-dynamo.slack.com

got my first Nvidia PR merged into main 🥳🥳🥳 dynamo#16736

EFA の同じノード内の GPU 間通信: libfabric の shm と SAR はじめに AWS の GPU インスタンスで EFA を使う通信ライブラリの多くは、libfabric という通信のライブラリを通して EFA を操作します。たとえば NCCL は、aws-ofi-nccl というプラグインを通して libfabric を使い、libfabric が rdma-core とドライバを通して EFA を動かします。この層の分担は「MoE の all-to-all を支える層: DeepEP、NCCL GIN、aws-ofi-nccl、EFA」で図にしています。この記事で扱う NIXL の LIBFABRIC バックエンドも、同じように libfabric を通して EFA を使います。 通信のライブラリ libfabric は、通信の方式をプロバイダという部品で切り替えます。AWS のネットワークカードである EFA を使うのが efa プロバイダで、同じノードの中で共有メモリを使うのが shm プロバイダです。efa プロバイダを使うと、相手が別のノードにいるときは EFA の装置を通ってデータが届きます。ところが、この記事で調べた libfabric 2.4.0 の efa プロバイダは、相手が同じノードにいるとき、既定の設定 (FI_EFA_ENABLE_SHM_TRANSFER=1) では EFA の装置を使わず、shm プロバイダにデータを任せます。shm プロバイダは、1 つの登録済みの GPU のバッファを送るときなど、条件がそろうと CUDA IPC を選びます。CUDA IPC は、別のプロセスの GPU のメモリを、自分のプロセスから開いてコピーできるようにする CUDA の仕組みです。このとき、受け取る側が送る側の GPU のメモリを開いてコピーします。CUDA IPC を選んだあとで、送る側がその準備に失敗すると、SAR という、ホストの共有メモリを経由して小分けに運ぶ方法に切り替えます。この記事では、この同じノードの中の経路を図で追います。 既定では、同じノードの相手へのデータは EFA の装置を通りません 図の左は、相手が別のノードにいる場合です。データは送る側の EFA の装置からネットワークを通り、相手の EFA の装置へ届きます。図の右は、相手が同じノードにいる場合です。efa プロバイダは相手が同じノードにいると判断すると、既定の FI_EFA_ENABLE_SHM_TRANSFER=1 では shm プロバイダにデータを任せます。FI_EFA_ENABLE_SHM_TRANSFER=0 のときは、同じノードの相手にも EFA の装置を使います。RDMA (Remote Direct Memory Access) の書き込みは、通信の装置が、相手のメモリへ直接データを書き込む操作です。測った構成では、この転送が装置の RDMA の書き込みとして出たことを、「直し方 1」の節で確かめます。 ここから、この記事で使う 3 つの言葉を、図で 1 つずつ説明します。 shm プロバイダは、ホストのメモリの領域で命令と応答を受け渡します shm プロバイダでは、各プロセスがホストのメモリに自分の領域を作ります。送る側は、相手が作った領域を shm_open で開いて mmap で自分のアドレス空間に対応付け、その領域にある命令の列 (command queue) に命令を書きます。受け取る側は、自分の領域に書かれた命令を読んで処理し、処理が終わったことを、送る側の領域にある応答の列 (response queue) で返します。受け取る側の領域には、あとで見る SAR で使う中継用の領域も入っています。データの中身がどう運ばれるかは、次の CUDA IPC と SAR で違います。 CUDA IPC では、受け取る側が送る側の GPU のバッファから 1 回でコピーします CUDA IPC (Inter-Process Communication) は、別のプロセスの GPU のメモリを使えるようにする CUDA の仕組みです。送る側は、自分の GPU のバッファから IPC ハンドルという識別子を作り、shm の命令に入れて渡します。この記事で扱う、GPU のバッファへの送信では、受け取る側がそのハンドルを開き、送る側の GPU のバッファから自分の GPU のバッファへ、1 回でコピーします。コピーをするのは、受け取る側のプロセスです。ハンドルを作る前に、送る側は cuMemGetAddressRange でバッファの場所を調べます。この呼び出しは、送る側のスレッドで有効な CUDA のコンテキスト (GPU ごとに作られ、その GPU で確保したメモリを管理するもの) の中でアドレスを探すので、そのコンテキストの中でバッファが見つからないとハンドルを作れません。この記事で見る失敗は、ここで起きます。手順の細かいところは「CUDA IPC による受け渡し」の節で見ます。 SAR では、ホストの共有メモリを中継して、1 かたまりにつき 2 回コピーします SAR (segmentation and reassembly) は、shm がデータを中継用の領域で小分けに運ぶ方法です。データの大きさや指定によって最初から選ばれることもあり、CUDA IPC を選んだあとでその準備に失敗したときの切り替え先にもなります。この記事で問題になるのは、後者の切り替えです。SAR では、データを決まった大きさ のかたまりに分け、送る側が自分の GPU のバッファから、受け取る側の共有メモリにある中継用の領域へコピーします。続いて受け取る側が、そこから自分の GPU のバッファへコピーします。中継用の領域は使い回し、データが全部渡るまでこれを繰り返します。CUDA IPC なら 1 回で済むコピーが、かたまりごとに 2 回になり、そのたびにホストのメモリを経由します。手順の細かいところは「SAR」の節で見ます。 同じノードの中のこの経路で、実際に何が起きていたのかを、次の節で問題として定めます。 調べたケース: 同じノード内の LIBFABRIC 転送 この記事では、SGLang で disaggregated inference (分離推論) をするときに、NIXL の LIBFABRIC バックエンドを使って、同じノードにいる相手へ KV cache を送るケースを調べました。SGLang の分離推論では、入力を読む prefill と、出力のトークンを 1 つずつ作る decode を、別々のプロセスに分けて動かします。prefill は、入力の各トークンについて計算した Attention の Key と Value を KV cache として保存し、decode へ送ります。decode はそれを使って、入力を計算し直さずに続きのトークンを作ります。その転送を受け持つのが、推論のプロセスのあいだでデータを転送するライブラリ NIXL です。この記事の構成では、prefill と decode は、それぞれモデルの計算を 2 枚の GPU に分けて受け持ちます。この分け方をテンソル並列 (TP) と呼び、GPU ごとに 1 つずつあるプロセスの番号を TP rank と呼びます。prefill と decode は別々のノードに置くのが基本の構成なので、同じノードに置くケースは、あまり想定されていないかもしれません。今回は、このケースで何が起きるかを確かめます。 ! SGLang の分離構成で NIXL の LIBFABRIC バックエンドを使い、prefill と decode が同じノードにあると、KV cache は EFA の装置ではなく libfabric の shm の経路を通ります。測った構成の既定の設定 (NIXL 1.1.0) では、この経路の転送が、shm を切ったときより大幅に遅くなりました。この記事は、その経路で何が起きているかを追います。 NIXL は、転送に使うバックエンドを選べます。この記事で扱うのは LIBFABRIC のバックエンドです。EFA のあるインスタンスで efa プロバイダを使う場合、送り先が同じノードにあれば、KV cache は上の shm の経路を通ります。NIXL には UCX のバックエンドもあります。UCX は libfabric を使わず、同じノードの相手には、UCX 自身の経路 (cuda_ipc や共有メモリなど) から選んで使います。そのため、この記事の話は、NIXL で LIBFABRIC のバックエンドを選んだときのものです。UCX のバックエンドでも、転送スレッドが選ぶ GPU と KV cache のある GPU が食い違う問題が起きるかどうかは、この記事では調べていません。 測った構成では、shm を使う既定の設定で、SGLang 自身が記録する KV 転送時間が、shm を使わない設定より大幅に長くなりました。値は最後の「4 つの組み合わせの値」の節に並べます。page size は、SGLang が KV cache を管理するページ 1 つに入るトークンの数で、この記事の数値はすべて page size 1 で測ったものです。この記事が答える問いは、次の 2 つです。 shm の経路のどこで、何が起きているのか それを避ける方法は何か 測った構成の shm の経路では、prefill の TP rank 1 (送る側) で、KV cache の書き込みを発行する SGLang の転送スレッドが、データのある 1 番の GPU ではなく 0 番の GPU を向いたまま、書き込みを発行していたと考えられます。CUDA は、いま使う GPU (current device) をスレッドごとに覚えていて、転送スレッドではこれを設定していません。そのため CUDA IPC の準備に失敗し、SAR に切り替わります。転送スレッドで GPU を指定すると、この失敗は起きなくなりました。この記事では、これを「問題」と呼びます。 この問題を避ける方法は、2 通りあります。 直し方 何を変えるか 1. shm を使わない設定 同じノードの相手にも、EFA の装置で RDMA の書き込みをする。shm の経路を使わない 2. NIXL 1.4.0 shm の経路のまま、書き込みの前に、メモリを登録したときのコンテキストにスレッドを合わせる。SAR に切り替わらない 次の表は、転送に関わる層を、上から呼び出す順に並べたものです。右の列が、この記事で見るところです。 層 受け持ち この記事で見るところ SGLang KV cache を持ち、転送を頼む 転送スレッドが GPU を選ばないこと NIXL の LIBFABRIC バックエンド 転送を libfabric の呼び出しに変える 送る前の CUDA のコンテキスト libfabric の efa 相手が同じノードなら shm に任せる shm を使う設定 libfabric の shm 共有メモリで渡す CUDA IPC と SAR 記事の中の「0 番」「1 番」は、各プロセスから見た GPU の番号を指します。prefill のプロセスには CUDA_VISIBLE_DEVICES=0,1、decode のプロセスには CUDA_VISIBLE_DEVICES=2,3 を設定していたので、たとえば decode の「0 番」はノードの 3 枚目の GPU です。この記事の数値は、次の条件で測ったものです。 調べた条件 項目 値 インスタンス g7.48xlarge 1 台。Kubernetes の 1 つの Pod に prefill と decode を入れ、それぞれテンソル並列の TP 2 で 2 枚の GPU を使う。Pod に割り当てた EFA の装置は 1 台で、prefill と decode が一緒に使う モデル DeepSeek-V2-Lite-Chat。revision は 85864749cd611b4353ce1decdb286193298f64c7 SGLang 0.5.12.post1 NIXL 1.1.0 と 1.4.0 libfabric 2.4.0 ルーター sglang-router 0.3.2、--pd-disaggregation --policy round_robin --disable-retries --request-timeout-secs 330 prefill と decode の共通の引数 --tp-size 2 --dtype bfloat16 --kv-cache-dtype bf16 --context-length 32768 --chunked-prefill-size 8192 --mem-fraction-static 0.75 --page-size 1 --enable-metrics --stream-interval 1 --disaggregation-transfer-backend nixl。prefix cache は有効のまま prefill だけの引数 --disaggregation-mode prefill --disaggregation-bootstrap-port 8998 decode だけの引数 --disaggregation-mode decode 環境変数 SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC、FI_PROVIDER=efa、FI_EFA_USE_DEVICE_RDMA=1、FI_LOG_LEVEL=info。shm を使う構成では FI_EFA_ENABLE_SHM_TRANSFER を設定せず既定の 1 のまま、shm を使わない構成では 0 要求 入力 8,192 トークン、出力 256 トークン。問題の警告の行は、起動したばかりのエンジンに要求を 1 つ送って数えた。KV 転送時間は、平均して 1 秒に 1 件の割合で 20 秒送って、構成ごとに 1 回測った。TTFT は、同じ負荷を、要求の届く間隔などをばらつかせる乱数の始まりの値 seed を 3 通りに変えて流し、合わせた 測ったもの SGLang が --enable-metrics で記録する段階ごとの時間。sglang:per_stage_req_latency_seconds の stage="prefill_transfer_kv_cache"。libfabric のログの警告の行。unable to use IPC for msg, fallback to using SAR と Failed to perform cuMemGetAddressRange 問題の警告は、手を加えていない SGLang の prefill と decode のサーバーを同じノードで起動し、ルーターを PD モードで立てて、長い入力を送るだけで出ます。KV 転送時間は、同じ長さの要求を、届く間隔をばらつかせながら平均して決まった割合で送り、測りました。 同じ構成を起動する手順 MODEL_PATH に、モデルを置いた場所か Hugging Face の名前を入れます。これは、上の条件の起動引数を並べたものです。このブロックは、上の条件と同じ版 (SGLang 0.5.12.post1、NIXL 1.1.0、libfabric 2.4.0) が入った、GPU 4 枚と EFA の装置 1 つを割り当てた 1 つの Pod の中で、そのまま流すと下に書いた数が出ることを、2026-10-20 に確かめています。最後の行で、起動したサーバーを止めます。 export MODEL_PATH=deepseek-ai/DeepSeek-V2-Lite-Chat export SGLANG_DISAGGREGATION_NIXL_BACKEND=LIBFABRIC export FI_PROVIDER=efa export FI_EFA_USE_DEVICE_RDMA=1 export FI_LOG_LEVEL=info COMMON="--model-path $MODEL_PATH --revision 85864749cd611b4353ce1decdb286193298f64c7 --host 127.0.0.1 --tp-size 2 --dtype bfloat16 --kv-cache-dtype bf16 --context-length 32768 --chunked-prefill-size 8192 --mem-fraction-static 0.75 --page-size 1 --enable-metrics --stream-interval 1 --disaggregation-transfer-backend nixl" CUDA_VISIBLE_DEVICES=0,1 python3 -m sglang.launch_server $COMMON --port 30000 --disaggregation-mode prefill --disaggregation-bootstrap-port 8998 > prefill.log 2>&1 & CUDA_VISIBLE_DEVICES=2,3 python3 -m sglang.launch_server $COMMON --port 30001 --disaggregation-mode decode > decode.log 2>&1 & until curl -sf 127.0.0.1 > /dev/null && curl -sf 127.0.0.1 > /dev/null; do sleep 10; done python3 -m sglang_router.launch_router --pd-disaggregation --prefill 127.0.0.1 8998 --decode 127.0.0.1 --host 127.0.0.1 --port 8000 --policy round_robin --disable-retries --request-timeout-secs 330 > router.log 2>&1 & until curl -sf 127.0.0.1 > /dev/null; do sleep 2; done python3 -c 'import json, random, urllib.request; ids = [random.randrange(1000, 30000) for _ in range(8192)]; req = urllib.request.Request("127.0.0.1", json.dumps({"input_ids": ids, "sampling_params": {"max_new_tokens": 256, "ignore_eos": True}}).encode(), {"content-type": "application/json"}); print(json.loads(urllib.request.urlopen(req, timeout=600).read())["meta_info"]["completion_tokens"])' grep -c "fallback to using SAR" prefill.log grep -c "cuMemGetAddressRange" prefill.log kill $(jobs -p) このブロックを流すと、生成されたトークンの数として 256 が出て、そのあとの 2 つの grep -c が、SAR への切り替えの警告と cuMemGetAddressRange の失敗の警告の行の数を出します。どちらも 0 より大きい数が出ます。shm を使わない構成を試すときは、サーバーを起動する前に export FI_EFA_ENABLE_SHM_TRANSFER=0 を足します。足して流すと、256 のあとに 0 と 0 が出ました。 以下では、次の「全体像」の節で層と経路の図を示したあと、efa から shm への受け渡し、CUDA IPC と SAR、問題 (送る側のスレッドの GPU)、直し方 1 の shm を使わない設定、直し方 2 の NIXL 1.4.0 の順に追い、最後に 4 つの組み合わせの値を並べます。 全体像 同じノードの相手へのデータは、既定では EFA の装置ではなく shm プロバイダを通ります SGLang は TP の rank ごとに 1 つのプロセスを立て、各プロセスは自分の GPU の上に KV cache を持ちます。KV cache の転送は NIXL が受け持ち、相手が同じノードにいるか別のノードにいるかで、libfabric の中の経路が分かれます。 NIXL は書き込みに、書き込みが終わったときに受け取る側へ一緒に届く小さな値を付けて発行します。この値を付けることを示す印が、libfabric の FI_REMOTE_CQ_DATA です。発行に使う関数は、1.1.0 では fi_writedata、1.4.0 では fi_writemsg です。この値の有無は、あとで見る shm の経路の選び方に関わります。 efa から shm への受け渡し 書き込みに印を付けるのは NIXL で、その印を見て経路を決めるのは libfabric です SGLang は KV cache の送信を NIXL に頼み、NIXL の LIBFABRIC バックエンドは、libfabric の書き込みの関数を FI_REMOTE_CQ_DATA の印付きで呼びます。その先の経路は libfabric が決めます。efa プロバイダは、相手が同じノードなら shm プロバイダに任せます。shm プロバイダは、印が付いているので送る側だけで書き終える CMA の経路を使わず、この記事の書き込みでは CUDA IPC か SAR で運びます。 efa プロバイダは、同じノードの相手との通信のために、shm プロバイダを中に抱えています。shm を使うかどうかは設定 enable_shm_transfer で決まり、既定値は 1 です。環境変数 FI_EFA_ENABLE_SHM_TRANSFER=0 でこれを 0 にすると、shm の経路そのものが作られません。この設定を使う直し方は、「直し方 1: shm を使わない設定」の節で見ます。 CUDA IPC による受け渡し shm の CUDA IPC では、データをコピーするのは受け取る側のプロセスです 別のプロセスの GPU のメモリは、そのアドレスを渡されただけでは使えません。CUDA IPC は、送る側が自分の GPU のメモリから IPC ハンドルという識別子を作り、受け取る側がそのハンドルを開いて、相手のメモリを自分のプロセスから使えるようにする仕組みです。 shm には、プロセス間のメモリコピー (CMA) を使って送る側だけで書き終える経路もあります。ただし、FI_REMOTE_CQ_DATA の付いた書き込みはこの経路を使いません。どの方法で渡すかは smr_select_proto が決め、送るバッファが 1 つで、登録済みの GPU のメモリにあり、相手も IPC を使えるときは、CUDA IPC を選びます。ここで登録とは、転送に使うメモリの領域を、通信のライブラリにあらかじめ知らせておく処理のことです。 細かい手順の前に、全体の流れを書きます。データは、送る側 (prefill) の GPU にある KV cache から、受け取る側 (decode) の GPU のバッファへ流れます。コピーを実行するのは、受け取る側のプロセスです。そのための通り道は、2 つの処理でできます。送る側が自分の GPU のバッファの IPC ハンドルを作って渡す処理と、受け取る側がそのハンドルを開いて送る側のバッファを読めるようにする処理です。ハンドルや完了の知らせは、ホストの共有メモリにある命令の列と応答の列でやり取りします。 図の番号の順に、1 回の書き込みで何が起きるかを見ます。libfabric では、完了を確かめる関数を呼んだときに、たまっている処理をその場で進めます。これを progress と呼びます。 送る側の転送スレッドが、cuMemGetAddressRange で、転送するポインタを含む確保領域の先頭と長さを調べます。この呼び出しは、そのスレッドでいま有効な CUDA のコンテキストの中でアドレスを探します。 その先頭について cudaIpcGetMemHandle で IPC ハンドルを作ります。ハンドルは確保領域の全体を指すので、転送するポインタの先頭からのずれ (オフセット) を一緒に記録します。 送る側は、ハンドル・オフセット・大きさを、受け取る側の領域にある命令の列で先に確保した場所に書き込みます。最後に送る側が命令を確定させると、受け取る側から見えるようになります。送る側は、データそのものはコピーしません。 受け取る側は progress の中でこの命令を読み、ハンドルを開いて、送る側の GPU のバッファを自分のプロセスから使えるようにします。開くのは cudaIpcOpenMemHandle で、開いた結果はキャッシュします。 受け取る側のスレッドが、開いたアドレスにオフセットを足した場所から自分のバッファへ、cudaMemcpy でコピーします。受け取る側は送る側の GPU のバッファから直接読み、SAR のように中継用の領域にいったん置くことはしません。 コピーの呼び出しが戻ってから、受け取る側は送る側の領域にある応答の列に処理が済んだことを書き、送る側は自分の progress でそれを見ます。送る側の書き込みが完了するのは、このときです。 1 か 2 が失敗すると、smr_do_ipc はこの書き込みを次の節の SAR に切り替えます。問題は、1 でスレッドのコンテキストの中にバッファが見つからないときに起きます。 SAR SAR では、データがホストの共有メモリを中継して 2 回コピーされます CUDA IPC を選んだあとで準備に失敗すると、shm は SAR という経路に切り替えます。SAR は segmentation and reassembly の略で、データを小さく分けて送り、受け取る側で組み立て直す方式です。切り替えるときの警告「unable to use IPC for msg, fallback to using SAR」は FI_WARN_ONCE で出るので、1 つのプロセスで 1 回しか出ません。 SAR では、受け取る側の共有メモリにある、1 つ 32 KiB の中継用の領域を使います。この領域をバウンスバッファと呼びます。送る側はメッセージごとに、空いているバウンスバッファを最大 64 個まで確保し、相手ごとの上限がある場合はその数までにします。 そのあと、2 つの手順をメッセージの終わりまで繰り返します。まず送る側が、自分の GPU のバッファから中継用の領域へデータをコピーし、データが入ったことを示すフラグを FULL にします。次に受け取る側が、そこから自分のバッファへコピーし、領域をまた使えることを示すフラグを EMPTY に戻します。GPU のメモリのデータは、ホストの共有メモリを中継する 2 段のコピーで、1 回に最大 2 MiB (64 × 32 KiB) ずつ運ばれます。SAR に切り替わることで KV 転送時間がどれだけ長くなるかは、次の節で、切り替えをなくしたときの値と比べます。 問題: 転送スレッドが向く GPU ! ここで、もとの目的に戻ります。分離推論では、prefill が作った KV cache を、decode へ渡します。この受け渡しにかかる時間は、最初のトークンが出るまでの時間 (TTFT) に入るからです。問題は、この受け渡しの送る側で起きます。prefill の TP rank 1 では、転送スレッドが別の GPU を向いていると考えられ、そのため CUDA IPC の準備に失敗し、ホストのメモリを中継する SAR に切り替わります。測った構成 (NIXL 1.1.0、shm を使う設定、page size 1、1 秒に 1 件) では、SGLang が記録する KV 転送時間の平均が、そのままで 1,886 ms、転送スレッドに正しい GPU を指定する 1 行を SGLang に足すと 461 ms でした。この 2 つは同じ回に測った組で、最後の節の表の値とは別の回の測定です。 問題: prefill の TP rank 1 では、転送スレッドが 0 番の GPU のコンテキストのまま、1 番の GPU のデータを送ろうとします CUDA の runtime API は、いま使う GPU をホストのスレッドごとに覚えていて、これを current device と呼びます。cudaSetDevice を呼ばないスレッドでは、0 番の GPU が既定になります。一方、cuMemGetAddressRange のような driver API は、そのスレッドで有効な CUDA のコンテキスト (context) の中でアドレスを探します。コンテキストは GPU ごとに作られ、その GPU で確保したメモリを管理します。 SGLang の各プロセスのメインのスレッドは、初期化のときに自分の GPU を選びます。一方、NIXL の転送を発行するスレッドは、あとから threading.Thread で作られます。このスレッドで動く transfer_worker は、GPU を選びません。 TP rank 0 の KV cache は 0 番の GPU にあるので、たまたま向きが合います。TP rank 1 では、転送スレッドが 0 番の GPU を向いたまま、1 番の GPU にある KV cache の書き込みを発行します。0 番のコンテキストの中では 1 番の GPU のアドレスが見つからず、ハンドルを作れないと考えられます。ただし、どの呼び出しで 0 番のコンテキストが有効になったかまでは確かめていません。 NIXL 1.1.0 で shm を使ったまま、起動したばかりのエンジンに 8,192 トークンの要求を 1 つ送ると、prefill の TP rank 1 のプロセスから、Failed to perform cuMemGetAddressRange: CUDA_ERROR_NOT_FOUND の警告と、SAR への切り替えの警告が出ます。切り替えた回数は、警告の行からは分かりません。回数は、最後の節で数えます。 SGLang の transfer_worker の最初に、torch.cuda.set_device(self.kv_args.gpu_id) を呼ぶ 1 行を足すと、同じ要求を送っても、どちらの警告も出ませんでした。 2026 年 10 月 19 日に確かめた時点で、この転送スレッドについての SGLang の Issue はありません。近い修正でマージ済みの sgl-project/sglang#38774 は、NIXL のバックエンドを作るスレッドで GPU を選ぶようにしたもので、main ブランチの transfer_worker は、まだ GPU を選んでいません。 直し方 1: shm を使わない設定 直し方 1 は、SGLang のプロセスを起動する前に、環境変数 FI_EFA_ENABLE_SHM_TRANSFER=0 を設定するものです。efa プロバイダは、同じノードの相手にも shm に任せず、EFA の装置で RDMA の書き込みをします。 この設定が効くのは、efa プロバイダの中に shm の経路そのものが作られないからです。問題の失敗は、shm が CUDA IPC の準備をするときに起きるので、shm を通らなければ、その準備も SAR への切り替えも起きません。SGLang の転送スレッドが GPU を選ばないことは、この設定でも変わりません。 shm を使わない設定で本当に EFA の装置を通るのかを、装置のカウンタで確かめました。rdma_write_bytes は、libfabric や NIXL の機能ではなく、EFA の装置が数えているカウンタです。装置が RDMA の書き込みで送ったバイト数を、Linux の EFA ドライバが統計の項目として登録し、カーネルの RDMA の共通部分 (RDMA core) が、それを sysfs のファイルとして見せます。ファイルの場所は、下の図のいちばん上に示しました。パスに infiniband とありますが、EFA は InfiniBand の装置ではありません。Linux の RDMA の仕組みはもともと InfiniBand のために作られたので、分類の名前が今も infiniband のままで、EFA のドライバも RDMA の装置としてここに登録されます。このファイルは装置が書き込むものではありません。読むたびに、カーネルが EFA ドライバを通して装置にその時点の値を問い合わせ、その値を返します。ただし、前の問い合わせから約 10 ms 以内に読むと、前の値をそのまま返します。 数えているのは EFA の装置で、EFA ドライバが登録し RDMA core が sysfs に出した値を、同じノードの別の Pod から読みました この記事の構成では、推論のエンジンを動かす Pod は、ホストのネットワークを使わない設定でした。この Pod の中からは、hw_counters のディレクトリが見えませんでした。同じノードに置いた、ホストのネットワークを使う Pod からは見えたので、その確認用の Pod から、負荷をかける前と後にすべてのカウンタを読んで差を取りました。負荷は、NIXL 1.1.0、page size 1 の構成に、平均して 1 秒に 1 件の割合で 15 秒のあいだ要求を送るものです。要求が届く間隔はばらつくため、この回は 15 件ではなく 17 件が届き、17 件すべてが完了しました。shm を使う既定の設定では、バイト数のカウンタはどれも 0 のままでした。shm を使わない設定では、rdma_write_bytes が 8,663,389,312 バイト増えました。 送るはずの KV cache の量は、モデルの設定から計算すると 8,663,334,912 バイトで、測った値との比は 1.0000063 です。負荷をかけない 20 秒にはカウンタが動かなかったことと合わせると、この測定の区間の KV cache の転送は、装置の RDMA の書き込みとして出たと読めます。差の 54,400 バイト (17 件で割ると 3,200 バイト) の内訳は分かっていません。 shm を使わない設定で起動したエンジンのログには、EFA will not use SHM for intranode communication because FI_EFA_ENABLE_SHM_TRANSFER=0 が出て、shm の行は 1 行も出ませんでした。shm の経路が作られないことそのものは、上で示したコードの分岐から言えることで、shm の中を直接見て確かめたわけではありません。 測り方と計算 項目 中身 数えたもの EFA の装置の hw_counters の全項目。rdma_write_bytes、rdma_write_recv_bytes、rdma_write_wrs、tx_bytes など 読んだ場所 同じノードに置いた、ホストのネットワークを使う確認用の Pod から sysfs を読んだ 負荷 入力 8,192 トークン・出力 256 トークンの要求を、平均 1 秒に 1 件の割合で 15 秒。届く間隔は指数分布でばらつき、この回は 17 件が届いて 17 件とも完了。NIXL 1.1.0、page size 1、TP 2 雑音の確認 どちらの設定でも、測る直前に負荷をかけない 20 秒のあいだ、カウンタはどの項目も動かなかった 送るはずの量 このモデルは MLA という Attention の方式で、KV を 1 層、1 トークンあたり 576 個の値に圧縮して持つ。576 は、圧縮した KV の 512 と位置の情報の 64 の和。1 トークンあたり 27 層 × 576 × 2 バイト = 31,104 バイト。1 件、1 rank で × 8,192 トークン = 254,803,968 バイト。MLA の KV は TP の 2 つの rank が同じものを持って送るので、× 2 rank × 17 件 = 8,663,334,912 バイト 使った装置 prefill と decode のどの rank も、Pod に割り当てた 1 台の装置を開き、その装置の中で折り返した。ノードのほかの装置のカウンタは 0。割り当てられる装置は Pod ごとに変わりうるので、ノードのすべての装置のカウンタを読んだ shm を使わない設定では、同じノードの中のほかの相手への libfabric の通信も、shm ではなく EFA の装置を通ります。 直し方 2: NIXL 1.4.0 直し方 2: NIXL 1.4.0 は、転送を発行するスレッドのコンテキストを、自分で合わせ直します NIXL 1.4.0 で問題が起きないのは、次の順に、TP rank 1 の転送スレッドが 1 番の GPU のコンテキストを向くようになるからです。 メモリを登録するとき、NIXL は KV cache のバッファを持っているコンテキストを調べて、エンジンに 1 つ記録します。TP rank 1 では、1 番の GPU のコンテキストです。 GPU のメモリから書き込みを出す前に、NIXL は、書き込みを出すスレッドを、記録したコンテキストに切り替えます (cuCtxSetCurrent)。これは既定の設定で、1 つのプロセスが 1 つの GPU だけを登録している場合の動きです。書き込みを出すのは、NIXL を呼んだ SGLang の転送スレッドです。 そのあと同じスレッドの上で、shm が IPC ハンドルを作る準備の中で、libfabric の cuMemGetAddressRange が呼ばれます。スレッドのコンテキストが KV cache のある GPU と合っているので、測った構成では、この呼び出しが失敗しなくなりました。 ハンドルの準備が失敗しないので、SAR への切り替えの分岐に入りません。 NIXL 1.1.0 も、登録のときに同じようにコンテキストを記録します。ただし、それを設定するのは登録の処理の中だけで、書き込みを出す前には設定し直しません。書き込みの前に設定し直す処理は、ai-dynamo/nixl#1581 の変更に含まれています。直ったのは NIXL の側で、SGLang の転送スレッドのコードは変わっておらず、自分では GPU を指定しないままです。 この動きは既定で有効です。ただし、登録のときに 1 つのプロセスが複数の GPU を使うと分かった場合は、代わりに書き込みのたびに cudaSetDevice で GPU を合わせます。 測った構成では、各プロセスは自分の GPU だけを登録するので、複数の GPU を使うという判定にはならず、ログの「Multi-GPU detected」は 0 行でした。働いたのは、登録時のコンテキストを設定し直す動きのほうです。これにより、SGLang を直さなくても、TP rank 1 でハンドルを作れるようになりました。実際に経路がどうなったかは、最後の節で数えます。 直し方 2 では、KV cache は shm の経路のまま、CUDA IPC で運ばれます。 4 つの組み合わせの値 最後に、NIXL の版と shm の設定の 4 つの組み合わせで測った値を並べます。元の構成は NIXL 1.1.0 で shm を使う設定、直し方 1 は NIXL 1.1.0 で shm を使わない設定、直し方 2 は NIXL 1.4.0 で shm を使う設定、両方は NIXL 1.4.0 で shm を使わない設定です。どれも page size 1 です。 組み合わせ KV 転送時間 ms 経路 TTFT の中央値 ms 元の構成 1,776 rank 0 が CUDA IPC、rank 1 が SAR 2,453 直し方 1 196 EFA 1,376 直し方 2 533 CUDA IPC 2,410 両方 267 EFA 1,389 負荷は、どれも平均して 1 秒に 1 件です。KV 転送時間は、SGLang が要求ごとに記録する prefill_transfer_kv_cache を、測ったあいだの全件で平均した値です。1 件ごとに、prefill 側が KV cache の送信を頼んでから、送り終わりを確認するまでの時間で、確認はスケジューラの1 周に 1 回なので、送り終わってから気づくまでの待ちも含みます。TTFT は、利用者から見た、要求を送ってから最初のトークンが返るまでの時間で、KV 転送時間はその一部です。表の TTFT は、20 秒の負荷を 3 つの seed で流して合わせた 66 件の中央値です。行ごとに別の回の測定です。直し方 2 では、KV 転送時間は元の構成より短い一方、TTFT の中央値はほぼ同じでした。その理由は、この記事では調べていません。 経路の列は、prefill から decode へ KV cache を運んだ経路です。表の時間を測った回とは別の回に、4 つの組み合わせのそれぞれで、NIXL の版と shm の設定だけを変え、page size 1、平均 1 秒に 1 件の負荷を 15 秒かけて判定しました。どの回も 17 件の要求が届き、計算した転送の量は「直し方 1」の節と同じです。判定には 3 つのものを使いました。1 つ目は EFA の装置のカウンタ rdma_write_bytes の増分で、計算した量との比を見ました。2 つ目は、送る側で IPC ハンドルを作る cudaIpcGetMemHandle と、受け取る側の CUDA IPC のコピーの回数です。3 つ目は、送る側の cuMemGetAddressRange の失敗と、SAR の中継用の領域を通るコピーの回数です。2 つ目と 3 つ目は、bpftrace で、プロセスごとに呼び出しを数えました。 元の構成では、装置の増分は 0 でした。prefill の rank 0 では cudaIpcGetMemHandle が 5,454 回呼ばれ、decode の rank 0 が CUDA IPC で 5,454 回、合わせて 4,331,667,456 バイトをコピーしました。これは rank 1 つ分の KV cache の量 (254,803,968 バイト × 17 件) と同じです。prefill の rank 1 では cuMemGetAddressRange が 5,454 回失敗し、中継用の領域を通るコピーが 136,782 回で、運んだ量は rank 0 と同じでした。SAR への切り替えの警告は、SAR の節で見たとおりプロセスごとに 1 回しか出ないので 1 行だけでしたが、切り替えが起きた回数は失敗の回数と同じ 5,454 回です。直し方 2 では、装置の増分は 0 で、prefill の rank 0 と rank 1 のどちらでも cudaIpcGetMemHandle が 1,998 回呼ばれ、decode の rank 0 と rank 1 がそれぞれ 1,998 回、rank 1 つ分の KV cache と同じ量を CUDA IPC でコピーしました。cuMemGetAddressRange の失敗と SAR への切り替えの警告は 0 行でした。直し方 1 の装置の増分は、「直し方 1」の節で見たとおりです。両方でも、装置の増分は直し方 1 と同じバイト数で、4 つのプロセスのどれでも IPC ハンドルの作成とコピーは 0 回で、SAR への切り替えの警告も 0 行、装置の RDMA の書き込みは 4,066 回でした。 まとめ この記事では、同じノードの相手へ KV cache を送るときに、libfabric の efa プロバイダと shm プロバイダがどう振る舞うかを紐解きました。efa プロバイダは既定で同じノードの相手を shm プロバイダに任せ、shm プロバイダは条件がそろうと CUDA IPC を選びます。送る側のスレッドのコンテキストにバッファが見つからないと SAR に切り替わり、FI_EFA_ENABLE_SHM_TRANSFER=0 では EFA の装置を通ります。流すデータの量や条件で結果は変わるので、この記事の数値は目安でしかありませんが、この記事の構成で、どの経路をいつ通るかという挙動は、4 つの組み合わせで確かめられました。

@kawaidgaf @NVIDIAAI Yep, you can run it on a single GPU too! There’s a quickstart guide here as well. ai-dynamo/dynamo That said, Dynamo really shines with multi-node, multi-GPU workloads, where it helps with request routing, SLA-driven scaling, and orchestration. For a single-GPU deployment, it’s worth weighing whether running an inference engine directly would be simpler for your use case, so make sure to weigh both options!:)

10./ NVIDIA AIPerf - Benchmarking Guides Measure first-token latency, token latency, throughput and goodput. Includes guides for controlled loads and realistic workload traces. ai-dynamo/aiperf Basic Python, PyTorch and Transformer knowledge will help. The learning materials are free to access. Running the examples may require paid compute.

Configuring Smart Routing in NVIDIA Dynamo NVIDIA Dynamo is an open-source, high-throughput, low-latency inference framework designed to deploy large-scale generative AI and reasoning models across multi-node, multi-GPU environments. Its KV Router intelligently directs inference requests by evaluating computational costs across workers, considering both prefill and decode phases. By maximizing KV cache reuse and minimizing redundant computation, smart routing reduces time-to-first-token (TTFT) by up to 40% and increases overall throughput by 25-30% compared to round-robin load balancing. This guide covers infrastructure setup, router enablement for both aggregated and disaggregated serving patterns, parameter tuning, and monitoring for production deployments. By the end, you'll have KV-aware smart routing configured and verified across both serving patterns, with the parameters tuned for your workload. --- Prerequisites Before you begin, ensure you: - Have access to a GPU-enabled server with NVIDIA GPUs installed and the NVIDIA Container Toolkit configured, using a non-root user with sudo privileges. - 2 GPUs for aggregated serving with smart routing. - 4 GPUs for disaggregated serving with smart routing. - Install Docker Engine and Docker Compose. - Create a Hugging Face account and generate an access token for gated models like Llama. - Understand basic LLM inference concepts (prefill vs decode phases, KV cache). Understanding KV-Aware Routing Traditional load balancing strategies like round-robin or random routing distribute requests evenly across workers without considering their cached state. KV-aware routing optimizes performance by directing requests to workers that already have relevant KV cache blocks, reducing redundant computation during the prefill phase. How KV Cache Routing Works The KV Router makes routing decisions based on a cost function that balances two factors. 1. Prefill Cost — the router calculates prefill cost by determining how many tokens need processing from scratch. When a worker has cached KV blocks matching the request's input prefix, those blocks can be reused, reducing prefill computation: Prefill blocks = (Total input tokens - Cached tokens) / Block size 2. Decode Cost — the router tracks active blocks on each worker to estimate decode capacity. Workers with fewer active sequences have more capacity for new requests. 3. Combined Cost Formula: cost = overlap_score_weight × prefill_blocks + decode_blocks Where: - overlap_score_weight balances TTFT optimization (higher values) vs load distribution (lower values). - Lower cost indicates a better routing choice. - Default overlap_score_weight is 1.0. Worker Selection Process The router evaluates all available workers and selects the one with the lowest cost. Consider this example with 3 workers and overlap_score_weight = 1.0: !KV Cache Router Worker Selection Process | Worker | Cached Blocks | Prefill Blocks | Decode Blocks | Total Cost | Selected | |--------|---------------|----------------|---------------|------------|----------| | Worker 1 | 2 | 8 | 10 | 18 (1.0 × 8 + 10) | No | | Worker 2 | 5 | 5 | 5 | 10 (1.0 × 5 + 5) | Yes | | Worker 3 | 8 | 2 | 9 | 11 (1.0 × 2 + 9) | No | Worker 2 provides the best balance of cache reuse and available capacity, minimizing both prefill computation and decode load. Performance Benefits KV-aware routing provides measurable improvements over basic load balancing: - TTFT Reduction: 30-40% lower time-to-first-token by maximizing cache hits. - Throughput Increase: 25-30% higher tokens/second by reducing redundant prefill computation. - Better GPU Utilization: More balanced load distribution across workers. - Cache Efficiency: Higher cache hit rates through intelligent request placement. These benefits are most significant for workloads with long input prompts (>2000 tokens) where prefill cost dominates, similar or repetitive input prefixes across requests, and multiple inference workers with independent KV caches. Clone the Dynamo Repository NVIDIA Dynamo provides deployment scripts, launch utilities, and runtime modules for inference workloads. 1. Clone the repository: $ git clone https://github.com/ai-dynamo/dynamo.git 2. Navigate to the repository directory: $ cd dynamo 3. Switch to the latest stable release: $ git checkout release/0.9.0 Visit the Dynamo releases page to find the latest stable version. Verify CUDA Version and Pull the Container Image The vLLM container requires a CUDA version match between the host driver and container runtime to prevent GPU kernel incompatibilities. 1. Check the installed CUDA version: $ nvidia-smi The output displays the CUDA version in the top-right corner of the table: +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.95.05 Driver Version: 580.95.05 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ .... 2. Pull the vLLM container image from NGC, matching the image tag to your system's CUDA version: For CUDA 13.x: $ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 For CUDA 12.x: $ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0 > Visit the NVIDIA NGC Catalog to view all available image tags and CUDA versions. 3. (Optional) Build the container from source instead of pulling the pre-built image: $ ./container/build.sh --framework VLLM This creates an image named dynamo:latest-vllm. If you use this locally built image, replace nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 with dynamo:latest-vllm in all subsequent commands. Configure Hugging Face Cache Permissions The container runs as UID 1000 and requires write access to the Hugging Face cache directory for model downloads. Incorrect permissions prevent the container from accessing cached model weights, causing worker initialization failures. 1. Create the cache directory if it does not exist: $ mkdir -p container/.cache/huggingface 2. Set ownership to the container user (UID 1000): $ sudo chown -R 1000:1000 container/.cache/huggingface 3. Set appropriate permissions: $ sudo chmod -R 775 container/.cache/huggingface Start Infrastructure Services Dynamo's KV Router requires etcd for worker discovery and NATS for KV cache event propagation. The etcd service maintains a registry of active workers and their capabilities, while NATS handles real-time KV block events (creation and eviction) that let the router track cached state across workers. 1. Start the infrastructure services using Docker Compose: $ docker compose -f deploy/docker-compose.yml up -d 2. Verify the services are running: $ docker compose -f deploy/docker-compose.yml ps The output displays the running etcd and NATS containers: NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS deploy-etcd-server-1 bitnamilegacy/etcd:3.6.1 "/opt/bitnami/script…" etcd-server 18 seconds ago Up 18 seconds 0.0.0.0:2379-2380->2379-2380/tcp, [::]:2379-2380->2379-2380/tcp deploy-nats-server-1 nats:2.11.4 "/nats-server -c /et…" nats-server 18 seconds ago Up 18 seconds 0.0.0.0:4222->4222/tcp, [::]:4222->4222/tcp, 0.0.0.0:6222->6222/tcp, [::]:6222->6222/tcp, 0.0.0.0:8222->8222/tcp, [::]:8222->8222/tcp KV Router Configuration Parameters The KV Router uses environment variables to control routing behavior, cache tracking, and worker selection logic. These parameters determine how the router balances Time To First Token (TTFT) optimization through cache reuse against Inter-Token Latency (ITL) through load distribution. All parameters can be set via environment variables or command-line arguments on the Dynamo frontend. Core Router Parameters | Variable | Description | Default | Recommended | Notes | |----------|-------------|---------|-------------|-------| | DYN_ROUTER_MODE | Enables KV cache-aware routing strategy | round_robin | kv | Set to kv to enable cache-aware routing. Other modes: round_robin, random | | DYN_ROUTER_TEMPERATURE | Controls routing randomness via softmax sampling | 0.0 | 0.1 | 0.0 = deterministic (always lowest cost). Higher values add randomness to prevent saturation | | DYN_KV_OVERLAP_SCORE_WEIGHT | Weight for prefix cache overlaps in cost calculation | 1.0 | 1.5 | Formula: cost = weight × prefill_blocks + decode_blocks. Higher values prioritize TTFT over ITL | | DYN_KV_CACHE_BLOCK_SIZE | KV cache block size in tokens (must match backend) | Backend-specific | 16 | Must match vLLM's block size. Used for prefill block calculation | | DYN_KV_EVENTS | Enables real-time KV event tracking from workers | true | true | When disabled, router predicts cache state with TTL-based expiration | Worker Capacity Thresholds These parameters define when a worker is considered "busy" and should be excluded from routing decisions. | Variable | Description | Default | Recommended | Notes | |----------|-------------|---------|-------------|-------| | DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD | Busy threshold based on KV cache block utilization | Not set | 0.85 | Value between 0.0-1.0. Workers exceeding this percentage are excluded. Can update via /busy_threshold API | | DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD | Busy threshold based on prefill token count | Not set | 10000 | Absolute token count. Workers exceeding this are excluded. Uses OR logic with threshold fraction | Infrastructure Configuration | Variable | Description | Default | Notes | |----------|-------------|---------|-------| | ETCD_ENDPOINTS | etcd server URL for worker discovery | N/A | Format: http://localhost:2379. Multiple endpoints separated by comma | | NATS_SERVER | NATS server URL for KV event propagation | N/A | Format: nats://localhost:4222. Required for KV event streaming | | DYN_EVENT_PLANE | Event plane backend for KV cache events | N/A | Set to nats to enable NATS-based event publishing | | PYTHONHASHSEED | Deterministic hash seed for KV event IDs | N/A | Set to 0 for reproducible KV event ID generation | | DYN_SYSTEM_PORT | Port for worker system status server | System-dependent | Used for health checks and metrics. Set per-worker to avoid conflicts (e.g., 8081, 8082) | Parameter Tuning Guidelines For TTFT Optimization (Low Latency): - Increase DYN_KV_OVERLAP_SCORE_WEIGHT to 1.5-2.0 (prioritize cache hits). - Lower busy thresholds: DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.70, DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=500. - Keep DYN_ROUTER_TEMPERATURE low (0.0-0.1) for deterministic selection. For Throughput Optimization (High Load): - Lower DYN_KV_OVERLAP_SCORE_WEIGHT to 0.5-1.0 (balance cache hits with load). - Higher busy thresholds: DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.95, DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=2000. - Increase DYN_ROUTER_TEMPERATURE to 0.3-0.5 for better load distribution. When KV Events Are Unavailable: - Set DYN_KV_EVENTS=false to use prediction-based routing. - Router tracks state via routing decisions with TTL-based expiration (default 120s). - Less accurate than real-time events but works without backend KV event support. Set Up the Deployment Environment Both aggregated and disaggregated serving deployments use the same KV routing configuration. This section prepares the container environment and creates a shared configuration file for all deployment scripts. 1. Export your Hugging Face token (replace YOUR_HF_TOKEN with your actual token): $ export HF_TOKEN=YOUR_HF_TOKEN 2. Run the vLLM container with GPU access and workspace mounting, using the image tag that matches your CUDA version: $ ./container/run.sh -it --framework VLLM --mount-workspace --image nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 -e HF_TOKEN=$HF_TOKEN This starts an interactive container session with GPU support and passes the Hugging Face token to the container. 3. Inside the container, create the common environment configuration file: $ cat << 'EOF' > ~/kv_routing.env # Infrastructure endpoints export ETCD_ENDPOINTS=http://localhost:2379 export NATS_SERVER=nats://localhost:4222 export DYN_EVENT_PLANE=nats # Set deterministic hash for KV event IDs export PYTHONHASHSEED=0 # KV Router Configuration export DYN_ROUTER_MODE=kv export DYN_ROUTER_TEMPERATURE=0.1 export DYN_KV_OVERLAP_SCORE_WEIGHT=1.5 export DYN_KV_CACHE_BLOCK_SIZE=16 export DYN_KV_EVENTS=true # Optional: Advanced routing configuration export DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.85 export DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=10000 echo "Environment configuration loaded:" echo " etcd: $ETCD_ENDPOINTS" echo " NATS: $NATS_SERVER" echo " Router Mode: $DYN_ROUTER_MODE" EOF This file contains shared configuration variables for all deployment scripts, including etcd endpoints, NATS server address, and KV routing parameters. Configure KV Routing for Aggregated Serving Aggregated serving combines prefill and decode phases on a single worker. KV-aware routing optimizes which worker receives each request based on cached KV blocks, improving performance even when workers handle both phases. Deploy Aggregated Serving with KV Routing The deployment script launches multiple vLLM workers with KV event publishing enabled and a frontend configured for KV-aware routing, handling process cleanup and providing routing configuration feedback. 1. Inside the container, create the aggregated serving deployment script: $ cat << 'EOF' > ~/nemotron_agg_kv_routing.sh #!/bin/bash source ~/kv_routing.env MODEL="${1:-nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1}" pkill -f "dynamo.frontend"; pkill -f "dynamo.vllm"; sleep 2 set -e trap 'kill 0' EXIT python -m dynamo.frontend --router-reset-states & sleep 10 # Worker 1 - GPU 0 (ZMQ port 5557) DYN_SYSTEM_PORT=${DYN_SYSTEM_PORT1:-8081} \ CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \ --model "$MODEL" \ --trust-remote-code \ --connector none \ --enforce-eager \ --enable-prefix-caching \ --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5557"}' & # Worker 2 - GPU 1 (ZMQ port 5558) DYN_SYSTEM_PORT=${DYN_SYSTEM_PORT2:-8082} \ CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm \ --model "$MODEL" \ --trust-remote-code \ --connector none \ --enforce-eager \ --enable-prefix-caching \ --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5558"}' & sleep 30 echo "Aggregated serving ready at http://localhost:8000 (Model: $MODEL)" wait EOF This script configures aggregated serving with KV-aware routing: - Accepts the model name as a command-line argument (defaults to NVIDIA Nemotron Nano 4B). - Sources the environment configuration created earlier (includes DYN_EVENT_PLANE=nats). - Uses --connector none for aggregated serving (prefill+decode on same worker). - Uses --enforce-eager for fast deployment without compilation. - Enables prefix caching with --enable-prefix-caching for KV cache reuse. - Enables KV event publishing with ZMQ endpoints (ports 5557, 5558) on both workers. - Configures system ports with DYN_SYSTEM_PORT to avoid conflicts (8081, 8082). - Uses --trust-remote-code to support NVIDIA Nemotron models. > KV Event Flow Architecture: vLLM natively publishes KV cache events via ZMQ. Dynamo's bridge component automatically translates these events to NATS for router consumption: > > ``` > ┌─────────────────────┐ > │ vLLM Worker 1 │ > │ (ZMQ: tcp://*:5557) │───┐ > └─────────────────────┘ │ ┌─────────────────────┐ > │ ┌──────────────┐ ┌──────────────┐ │ KV Router │ > ├───>│ Dynamo Bridge├───>│ NATS Server ├──>│ (Builds Radix Tree) │ > │ └──────────────┘ └──────────────┘ │ Global Cache View │ > ┌─────────────────────┐ │ └─────────────────────┘ > │ vLLM Worker 2 │ │ > │ (ZMQ: tcp://*:5558) │───┘ > └─────────────────────┘ > ``` > > This architecture provides the router with a unified view of cache state across all workers while allowing vLLM to use its native ZMQ event publishing. You'll see ZMQ endpoints in worker logs, but events flow through the Dynamo bridge to NATS for router consumption. 2. Make the script executable: $ chmod +x ~/nemotron_agg_kv_routing.sh 3. Run the aggregated serving script: $ ~/nemotron_agg_kv_routing.sh The script starts the frontend with KV routing enabled and launches two vLLM workers on GPUs 0 and 1. The frontend connects to NATS and begins consuming KV events from all registered workers, building a prefix (radix) tree to track cached blocks across workers. To deploy a different model, pass the model name as an argument: $ ~/nemotron_agg_kv_routing.sh "nvidia/Llama-3_3-Nemotron-Super-49B-v1_5" Verify KV Routing The router logs provide insights into worker selection and cost calculations. 1. Open a new terminal session and send a test request to the frontend: $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": [ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Explain the concept of KV cache in LLM inference."} ], "max_tokens": 100, "temperature": 0.7 }' The output displays the model's response in JSON format with streaming tokens. 2. Check the frontend logs for routing decision details, which show cost calculations for each worker: Formula for worker_id=7587893399646425890 dp_rank=0 with 0 cached blocks: 5.188 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.125 + 2.000 Formula for worker_id=7587893399646425892 dp_rank=0 with 0 cached blocks: 5.188 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.125 + 2.000 Selected worker: worker_id=7587893399646425892 dp_rank=0, logit: 5.188, cached blocks: 0, tree size: 0, total blocks: 75379 The router calculates cost per worker with cost = overlap_weight × prefill_blocks + decode_blocks. In this example, both workers have 0 cached blocks (cold start), equal prefill costs (2.125 blocks weighted by 1.5), and equal decode loads (2.000). With identical costs (5.188), the router selects a worker based on temperature sampling. 3. Send a second request with a similar prompt prefix: $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": [ {"role": "system", "content": "You are a helpful AI assistant."}, {"role": "user", "content": "Explain the concept of KV cache reuse in distributed LLM serving."} ], "max_tokens": 100, "temperature": 0.7 }' 4. Check the routing decision for the second request in the logs. If the router detects cached blocks from the previous request with a matching prompt prefix, the logs show a reduced prefill cost: Formula for worker_id=7587893399646425890 dp_rank=0 with 0 cached blocks: 5.375 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.250 + 2.000 Formula for worker_id=7587893399646425892 dp_rank=0 with 1 cached blocks: 3.875 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 1.250 + 2.000 Selected worker: worker_id=7587893399646425892 dp_rank=0, logit: 3.875, cached blocks: 1, tree size: 8, total blocks: 75379 The router prioritizes the worker with 1 cached block, reducing its cost from 5.375 to 3.875 — the cached block reduces prefill computation needed (1.250 blocks vs 2.250), demonstrating KV cache reuse resulting in faster TTFT for requests with overlapping prompt prefixes. 5. Open a new terminal session on the host (outside the container) and monitor the KV event stream to verify event publishing: $ docker run --rm -it --network container:deploy-nats-server-1 \ natsio/nats-box nats stream info namespace-dynamo-component-backend-kv-events The output displays the JetStream statistics including message count and consumer information: Information for Stream namespace-dynamo-component-backend-kv-events ...... Subjects: namespace-dynamo-component-backend-kv-events.* Replicas: 1 Storage: File ...... State: ...... Messages: 8 Bytes: 2.3 KiB Active Consumers: 1 ...... The message count increases as workers publish KV cache events — each message represents a KV cache state change, such as block creation or eviction. Configure KV Routing for Disaggregated Serving Disaggregated serving separates prefill and decode phases across different worker pools. The KV Router optimizes both prefill worker selection (based on cache hits) and decode worker assignment (based on load distribution), while coordinating KV cache transfer between phases via NIXL. Deploy Disaggregated Serving with KV Routing The deployment script configures the frontend with increased overlap weight for prefill optimization, launches prefill workers with low batch sizes for TTFT optimization, and decode workers with high batch sizes for throughput maximization. The NIXL connector enables efficient KV cache transfer between prefill and decode workers. 1. If the aggregated serving script is still running from the previous section, stop it by pressing Ctrl+C in the container terminal. 2. Inside the same container, create the disaggregated serving deployment script: $ cat << 'EOF' > ~/nemotron_disagg_kv_routing.sh #!/bin/bash source ~/kv_routing.env MODEL="${1:-nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1}" pkill -f "dynamo.frontend"; pkill -f "dynamo.vllm"; sleep 2 set -e trap 'kill 0' EXIT python -m dynamo.frontend --router-reset-states & sleep 10 # Decode Worker 1 - GPU 0 CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \ --model "$MODEL" \ --connector nixl \ --is-decode-worker \ --max-model-len 2048 \ --enable-prefix-caching \ --trust-remote-code & # Decode Worker 2 - GPU 1 VLLM_NIXL_SIDE_CHANNEL_PORT=20097 \ CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm \ --model "$MODEL" \ --connector nixl \ --is-decode-worker \ --max-model-len 2048 \ --enable-prefix-caching \ --trust-remote-code & # Prefill Worker 1 - GPU 2 (ZMQ port 5559) CUDA_VISIBLE_DEVICES=2 python3 -m dynamo.vllm \ --model "$MODEL" \ --connector nixl \ --is-prefill-worker \ --max-model-len 2048 \ --enable-prefix-caching \ --trust-remote-code \ --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5559"}' & # Prefill Worker 2 - GPU 3 (ZMQ port 5560) VLLM_NIXL_SIDE_CHANNEL_PORT=20099 \ CUDA_VISIBLE_DEVICES=3 python3 -m dynamo.vllm \ --model "$MODEL" \ --connector nixl \ --is-prefill-worker \ --max-model-len 2048 \ --enable-prefix-caching \ --trust-remote-code \ --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5560"}' & sleep 30 echo "Disaggregated serving ready at http://localhost:8000 (Model: $MODEL)" echo "Architecture: 2 Decode Workers (GPU 0-1) + 2 Prefill Workers (GPU 2-3)" wait EOF This script configures disaggregated serving with KV-aware routing and NIXL for KV cache transfer: - Accepts the model name as a command-line argument (defaults to NVIDIA Nemotron Nano 4B). - Sources the environment configuration created earlier (includes DYN_EVENT_PLANE=nats). - Uses --connector nixl to enable NIXL for KV cache transfer between prefill and decode workers. - Sets unique NIXL ports via VLLM_NIXL_SIDE_CHANNEL_PORT for workers 2 and 4 (20097, 20099). - Enables prefix caching on all workers with --enable-prefix-caching for KV cache reuse. - Configures decode workers on GPUs 0-1 to receive KV cache from prefill workers via NIXL. - Configures prefill workers on GPUs 2-3 with ZMQ endpoints (ports 5559, 5560) for KV event publishing. - Uses --trust-remote-code to support NVIDIA Nemotron models. - Uses --max-model-len 2048 for efficient memory usage across all workers. - Frontend automatically detects prefill and decode workers to enable disaggregated routing. 3. Make the script executable: $ chmod +x ~/nemotron_disagg_kv_routing.sh 4. Run the disaggregated serving script: $ ~/nemotron_disagg_kv_routing.sh The script starts the frontend, two decode workers on GPUs 0-1, and two prefill workers on GPUs 2-3. The frontend automatically detects prefill workers and creates an internal prefill router for cache-aware prefill worker selection, while the main router handles decode worker assignment based on load distribution. To deploy a different model, pass the model name as an argument: $ ~/nemotron_disagg_kv_routing.sh "nvidia/Llama-3_3-Nemotron-Super-49B-v1_5" Verify Disaggregated KV Routing Disaggregated routing involves two routing decisions: prefill worker selection based on cache overlap, and decode worker assignment based on load and available capacity. 1. Open a new terminal session and send a test request to verify the disaggregated flow: $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": [ {"role": "system", "content": "You are a helpful AI assistant specialized in distributed systems."}, {"role": "user", "content": "Explain how KV cache transfer works in disaggregated LLM serving using NIXL."} ], "max_tokens": 150, "temperature": 0.7 }' 2. Check the frontend logs for prefill routing decisions, which show prefill worker cost calculations with the higher overlap weight (1.5): INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425930 dp_rank=0 with 0 cached blocks: 4.125 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.750 + 0.000 INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425934 dp_rank=0 with 0 cached blocks: 4.125 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.750 + 0.000 INFO dynamo_llm::kv_router::scheduler: Selected worker: worker_id=7587893399646425930 dp_rank=0, logit: 4.125, cached blocks: 0, tree size: 0, total blocks: 75379 On the first (cold-start) request, both prefill workers have 0 cached blocks, resulting in equal costs (4.125). The decode blocks component is 0.0 because prefill workers only handle the prefill phase. 3. Check the decode routing decisions in the script output. After prefill completion, the decode router assigns the request to a decode worker: INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425936 dp_rank=0 with 0 cached blocks: 2.000 = 0.0 * prefill_blocks + decode_blocks = 0.0 * 2.750 + 2.000 INFO dynamo_llm::kv_router::scheduler: Selected worker: worker_id=7587893399646425936 dp_rank=0, logit: 2.000, cached blocks: 0, tree size: 0, total blocks: 75379 The prefill blocks component is 0.0 in this formula because decode workers don't compute prefills — they receive KV cache from prefill workers via NIXL transfer. 4. Monitor NIXL transfer success in the worker logs displayed by the script: INFO nixl_connector._nixl_handshake: NIXL compatibility check passed (hash: cae818f9461cb277c305be4b8cd4aca3273e21a6ea98f7399d1a339560e1b3ea) NIXL successfully transfers KV cache blocks from prefill to decode workers using RDMA for low-latency, high-bandwidth data transfer. 5. Send a second request with a similar prompt prefix to verify cache reuse: $ curl -X POST http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "messages": [ {"role": "system", "content": "You are a helpful AI assistant specialized in distributed systems."}, {"role": "user", "content": "Explain how KV cache reuse works in disaggregated LLM serving."} ], "max_tokens": 150, "temperature": 0.7 }' 6. Check the prefill routing logs for the second request, which show improved cache hits: INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425930 dp_rank=0 with 1 cached blocks: 2.250 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 1.500 + 0.000 INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425934 dp_rank=0 with 0 cached blocks: 3.750 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.500 + 0.000 INFO dynamo_llm::kv_router::scheduler: Selected worker: worker_id=7587893399646425930 dp_rank=0, logit: 2.250, cached blocks: 1, tree size: 2, total blocks: 75379 The prefill router now sees 1 cached block on the selected worker, reducing its cost from 3.750 to 2.250 — demonstrating KV-aware routing's ability to optimize TTFT by leveraging cached prompt prefixes. Configure KV Routing with SGLang and TensorRT-LLM The KV Router environment configuration is universal across all inference backends. The same environment variables used for vLLM deployments work identically with SGLang and TensorRT-LLM, requiring only backend-specific adjustments for block size and KV event publishing. Universal Environment Variables These environment variables work across all backends (vLLM, SGLang, TensorRT-LLM) without modification: # Infrastructure endpoints export ETCD_ENDPOINTS=http://localhost:2379 export NATS_SERVER=nats://localhost:4222 export DYN_EVENT_PLANE=nats export PYTHONHASHSEED=0 # Router configuration export DYN_ROUTER_MODE=kv export DYN_ROUTER_TEMPERATURE=0.1 export DYN_KV_OVERLAP_SCORE_WEIGHT=1.5 export DYN_KV_EVENTS=true # Load balancing thresholds export DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.85 export DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=10000 Backend-Specific Configuration While the router environment variables are universal, two aspects require backend-specific configuration. Block Size Configuration — DYN_KV_CACHE_BLOCK_SIZE must match your backend's actual block size: | Backend | Typical Block Size | Configuration | |---------|-------------------|---------------| | SGLang | 16 | export DYN_KV_CACHE_BLOCK_SIZE=16 | | TensorRT-LLM | 64 or 128 (varies by model) | export DYN_KV_CACHE_BLOCK_SIZE=64 | | vLLM | 16 | export DYN_KV_CACHE_BLOCK_SIZE=16 | > Mismatched block sizes between the router and workers cause incorrect cost calculations and routing decisions. Always verify your backend's block size configuration. KV Event Publishing Configuration — each backend uses different command-line parameters to enable KV event publishing to the router: | Backend | Parameter | Example Value | |---------|-----------|---------------| | SGLang | --kv-events-config | nats://localhost:4222 | | TensorRT-LLM | --publish-events-and-metrics | No value (flag only) | | vLLM | --kv-events-config | '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5557"}' | SGLang publishes events using a NATS URL format: --kv-events-config nats://localhost:4222 TensorRT-LLM uses a simple flag to enable event publishing: --publish-events-and-metrics vLLM uses a JSON configuration format with a ZMQ endpoint: --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5557"}' Tuning KV Router Parameters The KV Router provides several parameters to optimize routing behavior for different workload characteristics. Proper tuning balances cache reuse (TTFT optimization) with load distribution (ITL optimization). Overlap Score Weight The --kv-overlap-score-weight parameter controls the router's preference for cache hits versus load balancing. Default value: 1.0. Increase (1.5-2.0) for prefill-heavy workloads — long input prompts (>2000 tokens), short output generation (<200 tokens), and where minimizing TTFT is critical: $ python3 -m dynamo.frontend \ --router-mode kv \ --kv-overlap-score-weight 1.5 Decrease (0.5-0.8) for decode-heavy workloads — short input prompts (<500 tokens), long output generation (>1000 tokens), and where maximizing throughput is critical: $ python3 -m dynamo.frontend \ --router-mode kv \ --kv-overlap-score-weight 0.7 Set to 0.0 to disable cache-aware routing and use pure load balancing: $ python3 -m dynamo.frontend \ --router-mode kv \ --kv-overlap-score-weight 0.0 This mode ignores cached blocks and routes based solely on worker load — useful for workloads with no prefix overlap or for baseline performance comparisons. Router Temperature The --router-temperature parameter introduces randomness in worker selection to prevent saturation and improve load distribution. Default value: 0.0 (deterministic). - Temperature = 0.0 (default): Always select the lowest-cost worker. Best for maximizing cache reuse in stable workloads, with the risk of oversaturating workers with the best cache state. - Temperature = 0.2-0.5: Add slight randomness while still favoring low-cost workers. Best for production workloads with variable request patterns, giving better load distribution without sacrificing much cache efficiency: $ python3 -m dynamo.frontend \ --router-mode kv \ --router-temperature 0.3 - Temperature = 1.0+: More uniform distribution across workers. Best for debugging, testing, or when cache reuse is less important — approaches random selection at very high temperatures. KV Events Configuration The --no-kv-events flag disables real-time KV event tracking and switches to prediction-based routing. Default: KV events enabled. When enabled, the router predicts cache state based on its own routing decisions instead of receiving events from workers: $ python3 -m dynamo.frontend \ --router-mode kv \ --no-kv-events \ --router-ttl 120 \ --router-max-tree-size 1048576 \ --router-prune-target-ratio 0.8 Additional parameters for prediction mode: - --router-ttl 120: Expire cached blocks after 120 seconds (default: 120). - --router-max-tree-size 1048576: Maximum blocks before pruning (default: 2^20). - --router-prune-target-ratio 0.8: Prune to 80% of max size when threshold exceeded. Use prediction mode when your backend doesn't support KV event publishing, you want lower infrastructure overhead (no NATS dependency), or your deployment is simple enough that prediction accuracy is acceptable. When using --no-kv-events, configure backends to disable event publishing: - vLLM: Use --kv-events-config '{"enable_kv_cache_events": false}'. - SGLang: Do not use --kv-events-config. - TRT-LLM: Do not use --publish-events-and-metrics. Active Block Busy Detection The --active-decode-blocks-threshold parameter marks workers as busy when their KV cache utilization exceeds a threshold, preventing overload. Default: disabled (no threshold). $ python3 -m dynamo.frontend \ --router-mode kv \ --active-decode-blocks-threshold 0.85 \ --active-prefill-tokens-threshold 10000 - --active-decode-blocks-threshold 0.85: Mark worker busy at 85% KV cache utilization. - --active-prefill-tokens-threshold 10000: Mark worker busy when prefill tokens exceed 10,000. Busy workers are excluded from routing decisions until their load drops below the threshold, preventing request queueing and maintaining consistent latency. Thresholds can be updated at runtime without restarting the frontend: $ curl -X POST http://localhost:8000/busy_threshold \ -H "Content-Type: application/json" \ -d '{ "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "active_decode_blocks_threshold": 0.90, "active_prefill_tokens_threshold": 12000 }' The response confirms the updated thresholds: { "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1", "active_decode_blocks_threshold": 0.90, "active_prefill_tokens_threshold": 12000 } Router Replica Synchronization The --router-replica-sync flag enables active block state synchronization between multiple router replicas for consistent load balancing. Default: disabled. Enable it when launching multiple frontend replicas for fault tolerance and horizontal scaling: # Router replica 1 $ python3 -m dynamo.frontend \ --router-mode kv \ --http-port 8000 \ --router-replica-sync # Router replica 2 $ python3 -m dynamo.frontend \ --router-mode kv \ --http-port 8001 \ --router-replica-sync Each replica maintains its own view of active blocks (ephemeral state); replicas exchange routing decisions via NATS core messaging, and cached block state (persistent) syncs automatically via JetStream. New replicas start with zero active blocks but converge quickly through sync, giving consistent routing decisions and better load distribution across replicas. Monitor KV Router Performance To monitor KV Router performance metrics including routing decisions, cache efficiency, and worker utilization, deploy an observability stack (Grafana and Prometheus) alongside your Dynamo deployment. Add the KV Router environment variables from this guide to your deployment configuration, then use Grafana dashboards to visualize KV cache routing performance metrics and Prometheus to track TTFT improvements, cache hit rates, and worker selection patterns. Next Steps - Tune DYN_KV_OVERLAP_SCORE_WEIGHT and DYN_ROUTER_TEMPERATURE against your real traffic patterns rather than the defaults - Stand up Grafana and Prometheus dashboards to track TTFT, cache hit rate, and worker load over time - Test SGLang or TensorRT-LLM backends with the same router configuration to compare cache efficiency - Enable --router-replica-sync before scaling to multiple frontend replicas in production For the full guide with additional tips, visit the original article on Vultr Docs.

Qwen3.8 vs Ornith 1.5 on a Mac: speed and code review On a Mac Studio M4 Max with 64 GB, Qwen3.8-27B with MTP does not replace the 35B mixture-of-experts model I already run. Measured on 12 October 2026 under oMLX 0.7.0, Qwen generated 55 tokens per second against 90, and took 3.46 seconds to produce its first token on an 800-token prompt against 0.49. On a code review test the two caught the same bugs with reasoning off, and Qwen caught fewer with it on. The number that decides it is time to first token Qwen3.8-27B needs seven times longer than Ornith 1.5 to start answering an 800-token prompt: 3.46 seconds against 0.49, averaged over 8 requests per model. Output speed is closer, 55.1 tokens per second against 90.4. In a chat that gap is a nuisance. A coding agent or a code review sends long prompts many times in a row, and there the wait before the first token is most of the cost. oMLX is an inference server for Apple Silicon Macs that serves MLX models behind an OpenAI-compatible API. I run both models on it, and each one has its own trick for generating faster. Qwen3.8-27B is a dense model: all 27 billion parameters work on every token. It ships with MTP, multi-token prediction, an extra head trained with the model that drafts the next few tokens so the model can verify them in one pass. Ornith 1.5 35B-A3B is a mixture of experts: 35 billion parameters, about 3 billion active per token. I run it with DFlash, where a separate small draft model proposes a block of tokens and the main model checks them. The chart puts both measurements side by side. The first pair of bars is the one I care about. !Bar chart of two measurements on oMLX 0.7.0. Time to first token on an 800-token prompt: Ornith with DFlash 0.49 seconds, Qwen3.8-27B with MTP 3.46 seconds. Output speed: Ornith 90.4 tokens per second, Qwen 55.1. Ornith wins both. I measured with aiperf 0.12.0, one model loaded at a time, the server restarted before each run and no other traffic. This is the command. It works against any OpenAI-compatible server: aiperf profile --url http://localhost:8010 --model <model-id> \ --endpoint-type chat --streaming \ --concurrency 1 --request-count 8 --num-warmup-requests 2 \ --prompt-input-tokens-mean 800 --prompt-input-tokens-stddev 0 \ --output-tokens-mean 400 --output-tokens-stddev 0 \ --extra-inputs '{"temperature":0.6,"chat_template_kwargs":{"enable_thinking":false}}' \ --tokenizer <model-snapshot-dir> --random-seed 7 Upgrading oMLX helped Qwen by about 20%, and it was not enough Moving from oMLX 0.6.4 to 0.7.0 raised Qwen3.8-27B from 57-59 to 71.7 tokens per second on a code prompt and from 46-48 to 55-56 on prose, two runs per cell with 400 output tokens. The oMLX 0.7.0 release notes list faster MTP verify and decode for this model, and the numbers agree. Ornith stayed where it was: 103-121 tokens per second on code, 99-102 on prose. I also tried the obvious knob. MTP drafts up to 4 tokens per cycle by default on this model, and oMLX lets you raise the ceiling. At 6 the output speed was 53.9 tokens per second, at 8 it was 52.4, against 55.1 at the default. The adaptive controller already picks a depth that pays, so I put the setting back. On a code review test the two models tie With reasoning off, Qwen3.8-27B caught 9 of 20 planted bugs and Ornith 1.5 caught 8. With 20 cases, one bug of difference is noise. The test takes 20 real commits from four of my repositories, flips one operator on an added line (< to <=, == to !=, a dropped not) and asks the model to review the diff in a single call, without tools. Each case is paired with the clean diff, and a model only gets credit if it flags the planted line on the mutant and not on the clean version. | Model | Reasoning | Bugs caught | Unreadable answers | Findings per clean diff | Median time | |---|---|---|---|---|---| | Qwen3.8-27B + MTP | off | 9 of 20 (45%) | 4 of 40 | 0.7 | 10 s | | Ornith 1.5 + DFlash | off | 8 of 20 (40%) | 2 of 40 | 1.6 | 3 s | | Qwen3.8-27B + MTP | on | 6 of 20 (30%) | 12 of 40 | 0.3 | 194 s | | Ornith 1.5 + DFlash | on | 8 of 20 (40%) | 13 of 40 | 0.0 | 102 s | The Ornith rows were measured on 2 October 2026 on oMLX 0.6.4, the Qwen rows on 12 October on 0.7.0. With reasoning off, where they differ is false alarms. Qwen reports half as many problems on code that has none, and it takes three times as long per review. Turning reasoning on does not help Qwen. It caught 6 of 20 against 9 without it, and half of its 40 answers used up the whole 12,000-token output limit, which is why 12 of them never reached valid JSON. Ornith overthinks in the same way and loses 13 of 40 answers, but it holds at 8 of 20 and takes about half the time per review. A review that needs three minutes and fails to parse a third of the time is no use to me from either model. The two models do not fit in memory together Qwen3.8-27B takes 16.1 GB once loaded and Ornith 1.5 takes 19.8 GB. On a 64 GB machine that sounds fine, and under oMLX 0.6.4 both did load, with swap climbing to 15.3 of 16 GB. The memory guard in 0.7.0 refuses instead. While Qwen was busy, every request for Ornith got this: POST /v1/responses → 507: Cannot load Ornith-1.5-35B-A3B-MLX-4bit: projected memory 38.24GB would exceed the dynamic memory ceiling 34.51GB When Qwen is idle, oMLX evicts one model to load the other, and the log marks the unload as an immediate abort, so a request in flight on the evicted model is lost. I found out what that looks like under load by accident. I ran the review test on Qwen with reasoning on for 90 minutes while another tool kept sending requests to Ornith. oMLX loaded Ornith, evicted it six seconds later to make room for Qwen's prompt, loaded it again, and eventually evicted Qwen in the middle of an answer. Of Qwen's 40 requests, 19 failed. I had hoped to keep Qwen beside Ornith as a second opinion, and that is off the table. I have to pick one, and on these numbers I pick Ornith. Qwen sometimes cuts its own JSON, and the fix costs half the speed In 5 of 40 review answers Qwen3.8-27B returned {"findings":[] and stopped, with the closing brace missing. It only happened on empty answers of five or six tokens. I reran the five affected diffs three times each under four settings, with MTP on and off and with the presence penalty at 1.5 and 0. The cut answers appeared with MTP off too, and without the penalty, in up to 4 of 15 runs. As far as I can tell the model just ends early now and then. Constrained decoding removes it. oMLX has it built in through xgrammar, but the Homebrew formula leaves it out unless you ask: brew reinstall jundot/omlx/omlx --with-grammar With a JSON schema in the request, 0 of 30 answers were unparseable. The diagram shows what that costs. A grammar and MTP cannot run together, so those requests fall back to plain decoding at about 25 tokens per second. !Flow diagram of a review request to Qwen3.8-27B. Without a JSON schema it uses MTP decoding at about 55 tokens per second and up to 4 of 15 answers are unparseable. With a JSON schema it uses grammar-constrained decoding with MTP switched off at about 25 tokens per second and 0 of 30 answers are unparseable. What is still open Qwen's row with reasoning on mixes two runs. 21 answers come from the morning run described above, with other traffic on the server, and I repeated the 19 failed requests in the afternoon with everything else paused. I counted the morning answers because they were complete, but I did not rerun all 40 in one clean pass. The speed test used 8 requests and one prompt size. I did not test Qwen's vision, and I do not know what a JSON schema costs Ornith with DFlash. I removed Qwen from the proxy that evening and deleted its weights, so none of that will be measured. The next candidate is Muse Glimmer 30B, which is being measured now. For interactive work, where you wait on every answer, Ornith stays. If you are choosing a local model for a Mac, measure time to first token at the prompt length you really send before you look at tokens per second. For unattended jobs that gap matters much less.