Support for GRPO finetuning with vLLM

[AdityaKulshrestha]

Feature request

Hi team,

I wanted to enquire on the support for vLLM for GRPO finetuning. I tried with running the vllm instance using the commmand: trl vllm-serve --model Qwen/Qwen2-0.5B-Instruct

I got the this error message:

Calling add_step_closure function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
Calling iter_mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
[2025-09-21 12:21:36,921] [INFO] [real_accelerator.py:225:get_accelerator] Setting ds_accelerator to hpu (auto detect)
INFO 09-21 12:21:38 [__init__.py:254] Automatically detected platform hpu.
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/trl/scripts/vllm_serve.py", line 272, in llm_worker
    llm = LLM(
  File "/root/aditya/vllm-fork/vllm/entrypoints/llm.py", line 244, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/root/aditya/vllm-fork/vllm/engine/llm_engine.py", line 497, in from_engine_args
    vllm_config = engine_args.create_engine_config(usage_context)
  File "/root/aditya/vllm-fork/vllm/engine/arg_utils.py", line 1065, in create_engine_config
    model_config = self.create_model_config()
  File "/root/aditya/vllm-fork/vllm/engine/arg_utils.py", line 956, in create_model_config
    return ModelConfig(
  File "/usr/local/lib/python3.10/dist-packages/pydantic/_internal/_dataclasses.py", line 123, in __init__
    s.__pydantic_validator__.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
enforce_eager
  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]
    For further information visit https://errors.pydantic.dev/2.11/v/bool_type
INFO:     Started server process [3198502]
INFO:     Waiting for application startup.

Motivation

It would be easier training model requiring larger context length or large VRAM requirements by directly running a policy model on another card and serve it using vLLM.

Your contribution

I am happy to contribute to this PR, any sort of guidance would be really helpful.
I am more than happy to pick this task up.

[regisss]

Can you explain more how this relates to optimum-habana please?

[AdityaKulshrestha]

Optimum-habana is also maintaining trl for the fine-tuning support on Gaudi. The vLLM integration with trl allows trl to send updated weights to vLLM (policy model). I can't launch a pure vLLM instance since it requires the model's weights to be updated after the backpropagation.
Launching a vLLM instance through trl allows the trl to update the model weights during the training.

[AdityaKulshrestha]

I was able to run it with trl using trl vllm-serve Qwen/Qwen2.5-7B --enforce-eager 1. But faced this error during model training:

These are the logs of the vllm server.

INFO:     127.0.0.1:52792 - "POST /update_named_param/ HTTP/1.1" 200 OK
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.10/dist-packages/trl/scripts/vllm_serve.py", line 303, in llm_worker
    result = method(*args, **kwargs)
  File "/root/aditya/vllm-fork/vllm/entrypoints/llm.py", line 503, in collective_rpc
    return self.llm_engine.collective_rpc(method, timeout, args, kwargs)
  File "/root/aditya/vllm-fork/vllm/engine/llm_engine.py", line 2175, in collective_rpc
    return self.model_executor.collective_rpc(method, timeout, args,
  File "/root/aditya/vllm-fork/vllm/executor/uniproc_executor.py", line 59, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
  File "/root/aditya/vllm-fork/vllm/utils.py", line 2784, in run_method
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/trl/scripts/vllm_serve.py", line 136, in update_named_param
    self.model_runner.model.load_weights(weights=[(name, weight)])
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1943, in __getattr__
    raise AttributeError(
AttributeError: 'HpuModelAdapter' object has no attribute 'load_weights'

I guess I have to raise this issue on the vllm habana repo.

[regisss]

Optimum-habana is also maintaining trl for the fine-tuning support on Gaudi. The vLLM integration with trl allows trl to send updated weights to vLLM (policy model). I can't launch a pure vLLM instance since it requires the model's weights to be updated after the backpropagation. Launching a vLLM instance through trl allows the trl to update the model weights during the training.

Okay, got it. Happy to review any PR that can help you solve this 🙂
Our optimized TRL support is organized as follows:

• everything related to TRL is located in https://github.com/huggingface/optimum-habana/tree/main/optimum/habana/trl
• maybe more interesting for you, all our changes brought to TRL trainers are here: https://github.com/huggingface/optimum-habana/tree/main/optimum/habana/trl/trainer
  optimum/habana/trl basically follows the same tree structure as TRL, so feel free to add any file/folder inspired from TRL there.