Inferencing on Gaudi is failing

[dkiran1]

Hi,
I used the below code sample to infer the model on Gaudi cluster
import logging
from transformers import AutoTokenizer, AutoModelForCausalLM
from optimum.habana import GaudiTextGenerationPipeline, GaudiConfig

MODEL_NAME = "Qwen/Qwen3-4B-Thinking-2507"
DEVICE = "hpu"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=False)

model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
device_map=DEVICE,
trust_remote_code=True
)

gaudi_config = GaudiConfig(
use_hpu_graphs=True, # enable graph compilation for speed
use_kv_cache=True, # faster decoding
attention_op="gaudi_scaled_dot_product_attention"
)
pipe = GaudiTextGenerationPipeline(
model=model,
tokenizer=tokenizer,
gaudi_config=gaudi_config
)
prompt = "Explain the importance of space exploration in simple terms."
output = pipe(
prompt,
max_new_tokens=128,
batch_size=1,
do_sample=True,
top_p=0.9,
temperature=0.7
)
print("\n--- Generated Output ---\n")
print(output[0]["generated_text"])

Iam getting below error ImportError: cannot import name '_VALID_DICT_FIELDS' from 'transformers.training_args' (/usr/local/lib/python3.10/dist-packages/transformers/training_args.py)

Here is my config transformers :- 4.55.2, habana: 1.21.3.57_, optimum-habana 1.18.0 and optimum version 1.27.0 ,Can this be due to version mismatch? can you please provide the sample inferencing code on Gaudi?

[regisss]

optimum-habana v1.18 is not compatible with transformers v4.55. It requires transformers v4.51, please run:

pip install transformers==4.51.3

Besides, running

from optimum.habana import GaudiTextGenerationPipeline

will not work as GaudiTextGenerationPipeline is not defined in the source code of the library but in an example: https://github.com/huggingface/optimum-habana/blob/v1.18.1/examples/text-generation/text-generation-pipeline/pipeline.py#L12

[dkiran1]

If I use transformers v4.51, Iam getting the below error.
from transformers.training_args import (
ImportError: cannot import name '_VALID_DICT_FIELDS' from 'transformers.training_args' (/usr/local/lib/python3.10/dist-packages/tra
so I downgraded to 4.49.0. I used the example of pipeline
import argparse
import logging

from pipeline import GaudiTextGenerationPipeline
from run_generation import setup_parser
import time

Define a logger

logging.basicConfig(
format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
datefmt="%m/%d/%Y %H:%M:%S",
level=logging.INFO,
)
logger = logging.getLogger(name)

parser = argparse.ArgumentParser()
args = setup_parser(parser)

args.num_return_sequences = 1
#args.model_name_or_path = "Qwen/Qwen3-4B-Thinking-2507"
#args.model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
#args.model_name_or_path = "./merged_sft_model_phi_sql"
args.model_name_or_path = "microsoft/Phi-3.5-mini-instruct"
args.max_new_tokens = 1000
args.use_hpu_graphs = False
args.batch_size = 2
args.use_kv_cache = True
args.do_sample = True
#args.do_sample = False
print("Args value.. ",args)

pipe = GaudiTextGenerationPipeline(args, logger)
final_prompt = prompt.format(question=question)
output = pipe(final_prompt)

Its working ok with Qwen 2.5 based models , but giving below error for microsoft-phi3.3 models .
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: prob_dist must be 1 or 2 dim.
Is this model not supported yet? Do you have any fix

[regisss]

This should work with Transformers v4.51, can you paste here the whole error message?

Phi3 is not part of our list of validated models so it's not surprising that it doesn't work.

[dkiran1]

Is it ok to use 1.21.3 habana version with 1.18.0 ?
If I install Iam getting the warning...
"optimum-habana 1.18.0 requires transformers<4.50.0,>=4.49.0, but you have transformers 4.51.3 which is incompatible." Also this is the error that I get If I try to do inferencing
alling iter_mark_step function does not have any effect. It's lazy mode only functionality. (warning logged once)
[2025-08-25 08:59:46,009] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to hpu (auto detect)
Traceback (most recent call last):
File "/finetune/optimum-habana/examples/text-generation/text-generation-pipeline/sft_lora_soql_infernce.py", line 6, in
from run_generation import setup_parser
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/init.py", line 106, in wrapper
ret = original_fn(*args, **kwargs)
File "/finetune/optimum-habana/examples/text-generation/run_generation.py", line 33, in
from utils import (
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/init.py", line 106, in wrapper
ret = original_fn(*args, **kwargs)
File "/finetune/optimum-habana/examples/text-generation/utils.py", line 32, in
from optimum.habana.checkpoint_utils import (
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/init.py", line 106, in wrapper
ret = original_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/init.py", line 106, in wrapper
ret = original_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/init.py", line 19, in
from .transformers import (
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/init.py", line 106, in wrapper
ret = original_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/init.py", line 20, in
from .trainer import GaudiTrainer
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/init.py", line 106, in wrapper
ret = original_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/trainer.py", line 123, in
from .training_args import GaudiTrainingArguments
File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/core/init.py", line 106, in wrapper
ret = original_fn(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/training_args.py", line 40, in
from transformers.training_args import (
ImportError: cannot import name '_VALID_DICT_FIELDS' from 'transformers.training_args' (/usr/local/lib/python3.10/dist-packages/transformers/training_args.py)

I the link whatever you have shared I could see Phi in the inference list , do you mean Phi3 is not supported?

[regisss]

You can use optimum-habana 1.18.1 to have compatibility with transformers 4.51. But optimum-habana 1.18.0 should be compatible with synapse 1.21 too.

What is the output of pip show transformers?

Phi3 is a different model from Phi, and thus it relies on a different implementation.

[dkiran1]

Name: transformers
Version: 4.49.0
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: transformers@huggingface.co
License: Apache 2.0 License
Location: /usr/local/lib/python3.10/dist-packages
Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm
Required-by: optimum, optimum-habana, peft, sentence-transformers. This is teh output Iam seeing, Also I tried installing optimium-habana-1.18.1 with that I could install trasformer 4.51. But getting below error during inferencing

File "/finetune/optimum-habana/examples/text-generation/text-generation-pipeline/pipeline.py", line 61, in call
output = self.model.generate(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/generation/utils.py", line 1743, in generate
result = self._sample(
File "/usr/local/lib/python3.10/dist-packages/optimum/habana/transformers/generation/utils.py", line 2903, in _sample
torch.isin(input_ids[:, start_token_idx:], torch.tensor(eos_token_id)).int().argmax(dim=1)
RuntimeError: Expected all tensors to be on the HPU device, but found at least one input[idx=1] on cpu (details: torch.LongTensor)

[dkiran1]

The above issue was due to batch_size 2 , Now with optimum-hbana-1.18.1 is working with transformer 4.51. But one issue is If I try to inference Qwen3 model then Iam getting as below for any question.explanations,!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Qwen 3 model is there in supported list, can I please know what can be the reason for this

[regisss]

What is the code you ran to perform inference with Qwen3?

[dkiran1]

Its the same code which i have mentioned above
import argparse
import logging

from pipeline import GaudiTextGenerationPipeline
from run_generation import setup_parser
import time

Define a logger
logging.basicConfig(
format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
datefmt="%m/%d/%Y %H:%M:%S",
level=logging.INFO,
)
logger = logging.getLogger(name)

parser = argparse.ArgumentParser()
args = setup_parser(parser)

args.num_return_sequences = 1
#args.model_name_or_path = "Qwen/Qwen3-4B-Thinking-2507"
#args.model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
#args.model_name_or_path = "./merged_sft_model_phi_sql"
args.model_name_or_path = "microsoft/Phi-3.5-mini-instruct"
args.max_new_tokens = 1000
args.use_hpu_graphs = False
args.batch_size = 2
args.use_kv_cache = True
args.do_sample = True
#args.do_sample = False
print("Args value.. ",args)

pipe = GaudiTextGenerationPipeline(args, logger)
final_prompt = prompt.format(question=question)
output = pipe(final_prompt)

What I have observed we are seeing this issue on optimum-habana 1.18.1 for Qwen3 and other few models(fine tuned version of Qwen2.5) . If I downgrade the optimnum-habana to 1.18.0 , then its finedtuned version of Qwen2.5 worked out.

[regisss]

Which checkpoint of Qwen2.5 did you use?