File Access Error Preventing vLLM API Server from Starting

[Jimmy-Newtron]

Description

This issue is related to vllm-project/vllm#19192 in the vLLM project, where a file access error is preventing the vLLM API server from starting. This issue has been created to address specific aspects of the problem that are relevant to this project.

Context

In the main project, we encountered a file access error when trying to start the vLLM API server. The server fails to load the model weights from an S3 bucket, resulting in an initialization failure. The relevant details from the original issue are as follows:

• vLLM Version: 0.8.5
• Model Tag: s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8
• S3 Endpoint: https://s3.gra.io.cloud.ovh.net/
• Error Message: Exception: Could not send runai_request_with_credentials to libstreamer due to: b'File access error'

Problem Statement

Your project is responsible for handling S3 stream operations, and the file access error suggests that there might be an issue with how the S3 client is configured or how the streamer interacts with the S3 service. This issue aims to investigate and resolve the following:

1. S3 Client Configuration: Ensure that the S3 client is correctly configured with the necessary credentials and endpoint URL.
2. Streamer Interaction: Verify that the streamer correctly handles file requests and responses from the S3 service.
3. Error Handling: Improve error handling and logging to provide more detailed information about the file access error.

Expected Behavior

This project should correctly handle S3 stream operations, allowing the vLLM API server to load model weights from the specified S3 bucket without encountering file access errors.

Actual Behavior

Currently, the project fails to handle S3 stream operations correctly, resulting in a file access error when the vLLM API server attempts to load model weights.

Error Logs

INFO 06-05 01:31:37 [gpu_model_runner.py:1329] Starting to load model /tmp/tmpzndx51kq...
[2025-06-05 01:31:37] INFO credentials.py:1213: Found credentials in environment variables.
[2025-06-05 01:31:37] INFO configprovider.py:998: Found endpoint for s3 via: environment_global.
�[0;35m[2025-06-05 01:31:37.959] [DEBUG  ] [99 99] [streamer/impl/streamer/streamer.cc: 27 @ Streamer    ] Streamer concurrency 16 ; s3 block size 8388608 bytes;  ; file system block size 2097152 bytes;
[2025-06-05 01:31:37] INFO credentials.py:1213: Found credentials in environment variables.
�[0;34m[2025-06-05 01:31:37.965] [SPAM   ] [99 99] [streamer/impl/streamer/streamer.cc: 42 @ request     ] Requested to read 8 bytes from s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors offset 0
�[m�[0;34m[2025-06-05 01:31:37.965] [SPAM   ] [99 99] [streamer/impl/streamer/streamer.cc: 72 @ create_request] Requested to read asynchronously 8 bytes from s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors offset 0 in 1 chunks
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 99] [common/responder/responder.cc: 18 @ Responder        ] created responder for 1 running requests
�[m�[0;34m[2025-06-05 01:31:37.965] [SPAM   ] [99 99] [common/storage_uri/storage_uri.cc: 18 @ StorageUri   ] Checking: s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 99] [common/storage_uri/storage_uri.cc: 28 @ StorageUri   ] override url endpoint: https://s3.gra.io.cloud.ovh.net
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 99] [common/storage_uri/storage_uri.cc: 29 @ StorageUri   ] direct override of url endpoint in client configuration
�[m�[0;34m[2025-06-05 01:31:37.965] [SPAM   ] [99 99] [common/storage_uri/storage_uri.cc: 34 @ StorageUri   ] endpoint: https://s3.gra.io.cloud.ovh.net bucket: huggingface-hub path: neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 99] [streamer/impl/streamer/streamer.cc:117 @ create_request] Process file descriptors limit is 1048576 and concurrency level is 16
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 99] [streamer/impl/batches/batches.cc: 60 @ Batches       ] worker maximal range size is 8.39 MB
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 99] [streamer/impl/batches/batches.cc:140 @ handle_request] request file offset 0 size 8
�[m�[0;34m[2025-06-05 01:31:37.965] [SPAM   ] [99 99] [streamer/impl/batches/batches.cc:163 @ handle_request] task to read 8 bytes from file offset 0 to 8
�[m�[0;34m[2025-06-05 01:31:37.965] [SPAM   ] [99 99] [streamer/impl/batch/batch.cc: 66 @ Batch             ] Batch s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors range 0 - 8 ; 1 tasks
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 99] [streamer/impl/streamer/streamer.cc:151 @ create_request] sending 1 tasks to worker 0 total bytes 8 range 0 to 8
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 181] [streamer/impl/batch/batch.cc: 71 @ execute          ] Start reading from file s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 181] [utils/semver/semver.cc:119 @ get_glibc_version      ] GLIBC version is glibc 2.35
�[m�[0;35m[2025-06-05 01:31:37.965] [DEBUG  ] [99 181] [utils/semver/semver.cc:121 @ get_glibc_version      ] Semver = 2.35.0
�[m�[0;35m[2025-06-05 01:31:37.980] [DEBUG  ] [99 181] [s3/client_configuration/client_configuration.cc: 21 @ ClientConfiguration] S3 request timeout is set to 30000 ms
�[m�[0;35m[2025-06-05 01:31:37.980] [DEBUG  ] [99 181] [s3/client/client.cc:102 @ S3Client                  ] Using environment variable endpoint https://s3.gra.io.cloud.ovh.net , using configuration parameter endpointOverride
�[m�[0;35m[2025-06-05 01:31:37.980] [DEBUG  ] [99 181] [s3/client/client.cc:107 @ S3Client                  ] Setting s3 configuration useVirtualAddressing to 0
�[m�[0;35m[2025-06-05 01:31:37.980] [DEBUG  ] [99 181] [s3/client/client.cc:112 @ S3Client                  ] Setting s3 region to gra
�[m�[0;35m[2025-06-05 01:31:37.980] [DEBUG  ] [99 181] [s3/client/client.cc:123 @ S3Client                  ] Creating S3 client with given credentials
�[m�[0;35m[2025-06-05 01:31:37.993] [DEBUG  ] [99 181] [common/s3_wrapper/s3_wrapper.cc:107 @ create_client ] created client in 26 ms
�[m�[0;35m[2025-06-05 01:31:37.994] [DEBUG  ] [99 181] [common/responder/responder.cc: 18 @ Responder       ] created responder for 1 running requests
�[m�[0;34m[2025-06-05 01:31:37.994] [SPAM   ] [99 181] [s3/client/client.cc:195 @ async_read                ] Number of chunks is 1
�[m�[0;31m[2025-06-05 01:31:38.007] [ERROR  ] [99 209] [s3/client/client.cc:249 @ operator()                ] Failed to download s3 object :
�[m�[0;34m[2025-06-05 01:31:38.007] [SPAM   ] [99 209] [common/responder/responder.cc: 69 @ push            ] request index: 0 Response code:  response code: File access error ; 1 running requests
�[m�[0;34m[2025-06-05 01:31:38.007] [SPAM   ] [99 181] [common/responder/responder.cc: 48 @ pop             ] Sending response: request index: 0 Response code:  response code: File access error
�[m�[0;34m[2025-06-05 01:31:38.007] [SPAM   ] [99 181] [streamer/impl/batch/batch.cc:208 @ async_read       ] Received response index 0
�[m�[0;34m[2025-06-05 01:31:38.007] [SPAM   ] [99 181] [common/responder/responder.cc: 69 @ push            ] request index: 0 Response code:  response code: File access error ; 1 running requests
�[m�[0;35m[2025-06-05 01:31:38.007] [DEBUG  ] [99 181] [common/responder/responder.cc: 31 @ pop             ] responder stopped
�[m�[0;35m[2025-06-05 01:31:38.007] [DEBUG  ] [99 181] [streamer/impl/batch/batch.cc:204 @ async_read       ] Finished reading from file s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors - terminated
�[m�[0;34m[2025-06-05 01:31:38.008] [SPAM   ] [99 99] [common/responder/responder.cc: 48 @ pop              ] Sending response: request index: 0 Response code:  response code: File access error
�[m�[0;35m[2025-06-05 01:31:38.008] [DEBUG  ] [99 181] [streamer/impl/batch/batch.cc:116 @ execute          ] Terminated reading from file s3://huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors
�[m�[0;35m[2025-06-05 01:31:38.008] [DEBUG  ] [99 181] [./s3/client_mgr/client_mgr.h:153 @ push             ] Releasing S3 client

[RunAI Streamer] Overall time to stream 0 Bytes of all files: 0.05s, 0 Bytes/s
�[0;35m[2025-06-05 01:31:38.008] [DEBUG  ] [99 99] [streamer/impl/streamer/streamer.cc: 34 @ ~Streamer   ] Streamer shutting down
�[m�[0;35m[2025-06-05 01:31:38.008] [DEBUG  ] [99 99] [common/responder/responder.cc: 23 @ ~Responder       ] Responder shutdown
�[m�[0;35m[2025-06-05 01:31:38.008] [DEBUG  ] [99 99] [./s3/client_mgr/client_mgr.h:191 @ stop              ] Stopping all S3 clients
�[m�[0;35m[2025-06-05 01:31:38.010] [DEBUG  ] [99 99] [./s3/client_mgr/client_mgr.h:169 @ clear             ] Releasing all S3 clients
�[m�[0;35m[2025-06-05 01:31:38.010] [DEBUG  ] [99 99] [common/responder/responder.cc: 23 @ ~Responder       ] Responder shutdown

Stacktrace

�[mProcess EngineCore_0:
ERROR 06-05 01:31:38 [core.py:396] EngineCore failed to start.
ERROR 06-05 01:31:38 [core.py:396] Traceback (most recent call last):
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
ERROR 06-05 01:31:38 [core.py:396]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 06-05 01:31:38 [core.py:396]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 329, in __init__
ERROR 06-05 01:31:38 [core.py:396]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 64, in __init__
ERROR 06-05 01:31:38 [core.py:396]     self.model_executor = executor_class(vllm_config)
ERROR 06-05 01:31:38 [core.py:396]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
ERROR 06-05 01:31:38 [core.py:396]     self._init_executor()
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 47, in _init_executor
ERROR 06-05 01:31:38 [core.py:396]     self.collective_rpc("load_model")
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 06-05 01:31:38 [core.py:396]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 06-05 01:31:38 [core.py:396]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2456, in run_method
ERROR 06-05 01:31:38 [core.py:396]     return func(*args, **kwargs)
ERROR 06-05 01:31:38 [core.py:396]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 162, in load_model
ERROR 06-05 01:31:38 [core.py:396]     self.model_runner.load_model()
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1332, in load_model
ERROR 06-05 01:31:38 [core.py:396]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 06-05 01:31:38 [core.py:396]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
ERROR 06-05 01:31:38 [core.py:396]     return loader.load_model(vllm_config=vllm_config)
ERROR 06-05 01:31:38 [core.py:396]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 1508, in load_model
ERROR 06-05 01:31:38 [core.py:396]     model.load_weights(
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 576, in load_weights
ERROR 06-05 01:31:38 [core.py:396]     return loader.load_weights(
ERROR 06-05 01:31:38 [core.py:396]            ^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 261, in load_weights
ERROR 06-05 01:31:38 [core.py:396]     autoloaded_weights = set(self._load_module("", self.module, weights))
ERROR 06-05 01:31:38 [core.py:396]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 213, in _load_module
ERROR 06-05 01:31:38 [core.py:396]     for child_prefix, child_weights in self._groupby_prefix(weights):
ERROR 06-05 01:31:38 [core.py:396]                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 103, in _groupby_prefix
ERROR 06-05 01:31:38 [core.py:396]     for prefix, group in itertools.groupby(weights_by_parts,
ERROR 06-05 01:31:38 [core.py:396]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 101, in <genexpr>
ERROR 06-05 01:31:38 [core.py:396]     for weight_name, weight_data in weights)
ERROR 06-05 01:31:38 [core.py:396]                                     ^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 578, in <genexpr>
ERROR 06-05 01:31:38 [core.py:396]     for name, loaded_weight in weights)
ERROR 06-05 01:31:38 [core.py:396]                                ^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/weight_utils.py", line 459, in runai_safetensors_weights_iterator
ERROR 06-05 01:31:38 [core.py:396]     streamer.stream_file(st_file)
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/safetensors_streamer/safetensors_streamer.py", line 30, in stream_file
Traceback (most recent call last):
ERROR 06-05 01:31:38 [core.py:396]     safetensors_pytorch.prepare_request(self.file_streamer, path)
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/safetensors_streamer/safetensors_pytorch.py", line 101, in prepare_request
ERROR 06-05 01:31:38 [core.py:396]     safetensors_metadata = SafetensorsMetadata.from_file(fs, path)
ERROR 06-05 01:31:38 [core.py:396]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/safetensors_streamer/safetensors_pytorch.py", line 56, in from_file
ERROR 06-05 01:31:38 [core.py:396]     header_size_buffer = fs.read_file(filename, 0, SAFETENSORS_HEADER_BUFFER_SIZE)
ERROR 06-05 01:31:38 [core.py:396]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/file_streamer/file_streamer.py", line 77, in read_file
ERROR 06-05 01:31:38 [core.py:396]     runai_read(
ERROR 06-05 01:31:38 [core.py:396]   File "/usr/local/lib/python3.12/dist-packages/runai_model_streamer/libstreamer/libstreamer.py", line 47, in runai_read
ERROR 06-05 01:31:38 [core.py:396]     raise Exception(
ERROR 06-05 01:31:38 [core.py:396] Exception: Could not send runai_request_with_credentials to libstreamer due to: b'File access error'

Requested Assistance

Could you please help investigate and resolve the file access error? Any guidance on how to configure the S3 client, verify the streamer's interaction with the S3 service, or improve error handling would be greatly appreciated.

Thank you!

[noa-neria]

Thanks for reporting!

1. What is your version of runai-model-streamer?
2. Are you setting the aws region to be gra in some way in the config file or environment variable? If not, try to run with RUNAI_STREAMER_NO_BOTO3_SESSION=1

Please run with with RUNAI_STREAMER_S3_TRACE=1 for logs of the AWS S3 CPP SDK
Logs are written to a file in the directory where your application is running

[Jimmy-Newtron]

Thanks @noa-neria

1. I am using the runai_model_streamer==0.13.0 as per vLLM dependency.
2. I set the region using the environment variable.

Here an extract of the AWS S3 CPP SDK logs:

[TRACE] 2025-06-05 12:10:37.507 http-connection [140282448307776] id=0x7f93600179f0: Incoming message of size 555.
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Incoming response status: 400 (Bad Request).
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Incoming header: Content-Type: application/xml
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Incoming header: x-amz-id-2: tx2cc3e5248fa94d18bbe97-006841893d
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Incoming header: x-amz-request-id: tx2cc3e5248fa94d18bbe97-006841893d
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Incoming header: Date: Thu, 05 Jun 2025 12:10:37 GMT
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Incoming header: Transfer-Encoding: chunked
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Main header block done.
[TRACE] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Incoming body: 322 bytes received.
[TRACE] 2025-06-05 12:10:37.507 S3MetaRequest [140282448307776] id=0x7f93802b51a0 Incoming body for request 0x7f9378002160. Response status: 400. Data Size: 322. connection: 0x7f9378002e70.
[TRACE] 2025-06-05 12:10:37.507 S3MetaRequest [140282448307776] response body: 
<?xml version='1.0' encoding='UTF-8'?>
<Error><Code>InvalidURI</Code><Message>Couldn't parse the specified URI.</Message><RequestId>tx2cc3e5248fa94d18bbe97-006841893d</RequestId><Uri>https%3A//s3.gra.io.cloud.ovh.net/huggingface-hub/neuralmagic/Meta-Llama-3.1-8B-Instruct-FP8/model-00001-of-00002.safetensors</Uri></Error>

[TRACE] 2025-06-05 12:10:37.507 http-connection [140282448307776] id=0x7f93600179f0: Decoded 555 bytes of message, 0 bytes remain.
[TRACE] 2025-06-05 12:10:37.507 channel [140282448307776] id=0x7f9360002cd0: acquired message 0x7f936000b780 of capacity 16384 from pool 0x7f9360003490. Requested size was 18446744073709551060
[TRACE] 2025-06-05 12:10:37.507 tls-handler [140282448307776] id=0x7f9360014180: Bytes read 5
[TRACE] 2025-06-05 12:10:37.507 channel [140282448307776] id=0x7f9360002cd0: sending read message of size 5, from slot 0x7f93600140b0 to slot 0x7f9360017920 with handler 0x7f93600179f8.
[TRACE] 2025-06-05 12:10:37.507 http-connection [140282448307776] id=0x7f93600179f0: Incoming message of size 5.
[DEBUG] 2025-06-05 12:10:37.507 http-stream [140282448307776] id=0x7f9360018130: Client request complete, response status: 400 (Bad Request).
[DEBUG] 2025-06-05 12:10:37.507 S3MetaRequest [140282448307776] id=0x7f93802b51a0: Request 0x7f9378002160 finished with error code 14343 (aws-c-s3: AWS_ERROR_S3_INVALID_RESPONSE_STATUS, Invalid response status from request) and response status 400
[ERROR] 2025-06-05 12:10:37.507 S3MetaRequest [140282448307776] id=0x7f93802b51a0 Meta request cannot recover from error 14343 (Invalid response status from request). (request=0x7f9378002160, response status=400)

[noa-neria]

The server returns bad request, very similar to #46
This is most probably an issue in the storage backend

[gilljon]

x-posting from #46

I'm still getting this issue with the following env vars, could someone help me?

• name: TORCHINDUCTOR_CACHE_DIR
  value: /.cache
• name: DO_NOT_TRACK
  value: '1'
• name: VLLM_NO_USAGE_STATS
  value: '1'
• name: AWS_USE_FIPS_ENDPOINT
  value: "true"
• name: AWS_STS_REGIONAL_ENDPOINTS
  value: "regional"
• name: AWS_ENDPOINT_URL_S3
  value: "https://s3-fips.us-gov-west-1.amazonaws.com/"
• name: AWS_ENDPOINT_URL
  value: "https://s3-fips.us-gov-west-1.amazonaws.com/"
• name: AWS_ROLE_ARN
  value: "arn:aws-us-gov:iam::"
• name: AWS_SDK_LOAD_CONFIG
  value: "true"
• name: AWS_REGION
  value: "us-gov-west-1"
• name: AWS_DEFAULT_REGION
  value: "us-gov-west-1"
• name: AWS_WEB_IDENTITY_TOKEN_FILE
  value: "/var/run/secrets/kubernetes.io/serviceaccount/token"
• name: RUNAI_STREAMER_NO_BOTO3_SESSION
  value: "1"
• name: AWS_EC2_METADATA_DISABLED
  value: "true"
  Also seeing

Exception: Could not send runai_request_with_credentials to libstreamer due to: b'File access error'