[Bug]: DP request distribution becomes imbalanced under long-context workload on H20 GPU

[Cancan0320]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.10.0-182.0.0.95.r1941_123.hce2.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   :
GPU models and configuration :
GPU 0: NVIDIA H20
GPU 1: NVIDIA H20
GPU 2: NVIDIA H20
GPU 3: NVIDIA H20
GPU 4: NVIDIA H20
GPU 5: NVIDIA H20
GPU 6: NVIDIA H20
GPU 7: NVIDIA H20

Nvidia driver version        : 550.54.15
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               176
On-line CPU(s) list:                  0-175
Vendor ID:                            GenuineIntel
BIOS Vendor ID:                       Intel(R) Corporation
Model name:                           Intel(R) Xeon(R) Platinum 8458P
BIOS Model name:                      Intel(R) Xeon(R) Platinum 8458P
CPU family:                           6
Model:                                143
Thread(s) per core:                   2
Core(s) per socket:                   44
Socket(s):                            2
Stepping:                             8
BogoMIPS:                             5400.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves xfd cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts hfi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization:                       VT-x
L1d cache:                            4.1 MiB (88 instances)
L1i cache:                            2.8 MiB (88 instances)
L2 cache:                             176 MiB (88 instances)
L3 cache:                             165 MiB (2 instances)
NUMA node(s):                         2
NUMA node0 CPU(s):                    0-43,88-131
NUMA node1 CPU(s):                    44-87,132-175
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Vulnerable: eIBRS with unprivileged eBPF
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.6
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu129
[pip3] torchvision==0.25.0+cu129
[pip3] transformers==4.57.6
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.19.0
vLLM Build Flags:
  CUDA Archs: 7.0 7.5 8.0 8.9 9.0 10.0 12.0; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity       NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     PIX     SYS     SYS     SYS     SYS     SYS     SYS     0-43,88-131        0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    SYS     SYS     PIX     PIX     SYS     SYS     SYS     SYS     0-43,88-131        0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-43,88-131        0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     0-43,88-131        0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     PIX     SYS     SYS     44-87,132-175      1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     PIX     PIX     44-87,132-175      1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     44-87,132-175      1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     44-87,132-175      1               N/A
NIC0    PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      PIX     SYS     SYS     SYS     SYS     SYS     SYS
NIC1    PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX      X      SYS     SYS     SYS     SYS     SYS     SYS
NIC2    SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      PIX     SYS     SYS     SYS     SYS
NIC3    SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX      X      SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      PIX     SYS     SYS
NIC5    SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX      X      SYS     SYS
NIC6    SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      PIX
NIC7    SYS     SYS     SYS     SYS     SYS     PIX     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PIX      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
TORCH_CUDA_ARCH_LIST=7.0 7.5 8.0 8.9 9.0 10.0 12.0
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.9.1
VLLM_ENABLE_CUDA_COMPATIBILITY=0
LD_LIBRARY_PATH=/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

🐛 Describe the bug

Bug Title

H20 GPU: Imbalanced request distribution across vLLM DP engines under long-context workloads

Issue Description

We observed a significant difference in vLLM Data Parallel request distribution behavior between short-context and long-context workloads on H20 GPU.

For short-input / short-output workloads, such as GSM8K-style data with around 1k input tokens and 1k output tokens, DP request distribution looks generally normal. Requests can be distributed across multiple DP engines, and we did not observe long-term request accumulation on a single engine.

However, when switching to a long-context workload using LongBench narrativeqa, DP request distribution becomes noticeably imbalanced. At the beginning of the test, requests are distributed relatively evenly. After the workload runs for a period of time, most requests gradually concentrate on engine=0, while other engines receive significantly fewer requests.

Although the issue does not completely collapse into only one active engine on H20 GPU, the imbalance is clearly visible from vLLM metrics.

This suggests that under long-context workloads, vLLM DP scheduling may not be fully aware of the actual load of each engine. This is especially noticeable when prompt lengths vary significantly and each request consumes much more prefill time and KV cache.

Test Environment

Hardware: H20 GPU
Inference framework: vLLM
Model: Qwen3-4B
Deployment mode: Data Parallel, multiple DP engines
Short-context workload: GSM8K-style data, around 1k input tokens and 1k output tokens
Long-context workload: LongBench narrativeqa
Concurrency: 32
Long-context output length: fixed around 5k tokens

Expected Behavior

Under DP deployment, requests should be distributed relatively evenly across multiple DP engines, or at least scheduled based on the actual load of each engine.

Expected behavior includes:

1. When multiple engines are available, new requests should not be continuously assigned to the same engine while other engines are relatively idle.
2. Long-running requests should not cause persistent load imbalance across DP engines.
3. DP scheduling should consider the actual pressure of each engine, such as running requests, waiting requests, KV cache usage, prompt token length, or token-level workload, rather than relying only on request count or local scheduling state.

Actual Behavior

Under the LongBench narrativeqa long-context workload, request distribution across DP engines becomes noticeably imbalanced.

Observed behavior:

1. At the beginning of the test, request distribution is relatively balanced.
2. After the test runs for a while, requests gradually concentrate on engine=0.
3. engine=1 and engine=2 may still receive some requests, but significantly fewer than engine=0.
4. Some engines remain lightly loaded for certain periods.
5. Near the end of the test, request distribution may become relatively balanced again, but the imbalance in the middle stage remains obvious.

Reproduction Steps

1. Start vLLM with Qwen3-4B on H20 GPU and enable Data Parallel with multiple DP engines.

2. First run a short-input / short-output workload, such as GSM8K-style data:

• Around 1k input tokens
• Around 1k output tokens
• Around 1024 requests
• Same DP configuration

3. Monitor request distribution across DP engines using vLLM metrics:

curl http://localhost:8004/metrics 2>/dev/null | grep "vllm:num_requests"

Focus on the following metrics:

vllm:num_requests_running
vllm:num_requests_waiting

4. Observe that under the short-context workload, requests are generally distributed across multiple DP engines, and no obvious long-term imbalance is observed.

5. Then switch to the LongBench narrativeqa long-context workload:

• Concurrency: 32
• Output length: fixed around 5k tokens
• Prompt length varies significantly, from around 8k tokens to more than 60k tokens

6. Use a script to continuously collect running and waiting metrics for each DP engine every few seconds.

7. Plot num_requests_running and num_requests_waiting for each engine over time to observe how DP engine distribution changes.

Observations

Since I am currently unable to upload complete images or files, I am pasting part of the vLLM metrics logs here to illustrate the issue.

At the beginning of the service, the request distribution across engines is relatively even:

2026-05-28 12:53:45
vllm:num_requests_running{engine="0",model_name="qwen3_4b_pcy"} 8.0
vllm:num_requests_running{engine="1",model_name="qwen3_4b_pcy"} 8.0
vllm:num_requests_running{engine="2",model_name="qwen3_4b_pcy"} 8.0
vllm:num_requests_running{engine="3",model_name="qwen3_4b_pcy"} 8.0
vllm:num_requests_waiting{engine="0",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="1",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="2",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="3",model_name="qwen3_4b_pcy"} 0.0

In the middle stage of the evaluation, requests become heavily concentrated on engine=0:

2026-05-28 13:00:26
vllm:num_requests_running{engine="0",model_name="qwen3_4b_pcy"} 27.0
vllm:num_requests_running{engine="1",model_name="qwen3_4b_pcy"} 5.0
vllm:num_requests_running{engine="2",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_running{engine="3",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="0",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="1",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="2",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="3",model_name="qwen3_4b_pcy"} 0.0

Near the end of the evaluation, the distribution becomes relatively more balanced again:

2026-05-28 13:06:57
vllm:num_requests_running{engine="0",model_name="qwen3_4b_pcy"} 14.0
vllm:num_requests_running{engine="1",model_name="qwen3_4b_pcy"} 4.0
vllm:num_requests_running{engine="2",model_name="qwen3_4b_pcy"} 3.0
vllm:num_requests_running{engine="3",model_name="qwen3_4b_pcy"} 10.0
vllm:num_requests_waiting{engine="0",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="1",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="2",model_name="qwen3_4b_pcy"} 0.0
vllm:num_requests_waiting{engine="3",model_name="qwen3_4b_pcy"} 0.0

From the metrics above, DP request distribution on H20 GPU is not always completely imbalanced. Instead, it shows a stage-wise imbalance pattern under long-context workloads: the distribution is relatively balanced at the beginning, becomes clearly concentrated on engine=0 in the middle stage, and may recover to a more balanced state near the end.

This behavior is different from the short-context GSM8K-style workload, where we did not observe the same degree of DP distribution imbalance.

From the collected metrics, the DP behavior under short-context and long-context workloads is clearly different.

For the short-context GSM8K-style workload:

1. DP distribution is mostly balanced.
2. Each engine can receive requests.
3. No obvious long-term request concentration or waiting accumulation is observed.

For the LongBench narrativeqa long-context workload:

1. Requests are relatively balanced at the beginning.
2. In the middle stage of the test, many requests are concentrated on engine=0.
3. Other engines receive significantly fewer requests, and some engines have almost no running requests for a period of time.
4. Near the end, request distribution may become relatively balanced again.

The input token distribution of LongBench narrativeqa is not strictly monotonically increasing. It fluctuates significantly, with short and long samples interleaved. In our test, the approximate statistics are:

min input tokens: ~8,378
max input tokens: ~65,408
average input tokens: ~31,398
median input tokens: ~29,944

Therefore, the imbalance does not seem to be caused simply by monotonically increasing input length. Instead, it appears to be related to long-context workload characteristics in general, including long prompts, high KV cache pressure, and large variance in request cost.

Possible Cause

This behavior may indicate that under long-context workloads, vLLM DP scheduling is not fully aware of the actual workload pressure on each engine.

In long-context inference, the number of running requests alone cannot accurately represent the real load of an engine. For example:

engine=0: running=2, but each request has ~60k prompt tokens
engine=1: running=2, but each request has ~2k prompt tokens

Both engines have the same number of running requests, but their actual prefill cost, KV cache usage, and execution time can be very different.

If DP scheduling mainly considers request count, queue state, local engine state, or scheduling timing, but does not consider token-level workload, KV cache pressure, or estimated prefill cost, then long-context workloads can easily amplify small scheduling imbalance.

Questions for the vLLM Team

Could the vLLM team help confirm the following points?

1. How does vLLM currently select a DP engine for a new request?
2. Does the DP scheduler consider per-engine running requests, waiting requests, KV cache usage, or token-level workload?
3. Is there any existing configuration to enable more load-aware DP request routing?
4. Would it be possible to improve DP scheduling under long-context workloads by considering prompt length, estimated prefill cost, or KV cache pressure?

Additional Notes

This issue was observed when using vLLM DP on H20 GPU under long-context workloads.

The short-context GSM8K-style workload does not show the same degree of imbalance, which suggests that this issue is more strongly related to long-context request characteristics rather than DP deployment itself being unusable.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.