Unable to read MinIO S3 credentials from helm chart

[agkjwork]

Hello, im kinda new to this so pardon my ignorance. I am currently attempting to deploy gemma-4-31B-it model with vLLM 0.23.0 using vllm production stack helm chart and attempting to use MinIO S3 to stream the tensor and config files however im constantly getting ValueError: Could not receive runai_response from libstreamer due to: b 'File access error'

I am unable to paste the full helm chart and error logs as im working in a locked down environment but hopefully these few settings should paint a picture.

modelSpec:
  - name: "gemma-4-31B-it"
    modelUrl: s3://my-bucket/models/google/gemma-4-31B-it
      ......
    vllmConfig:
      enabledChunkedPrefil: true
        ...........
      extraArgs: ["--load-format", "runai_streamer".............]
    env:
      - name: SSL_CERT_FILE
        value: /certs/ca.crt

      - name: AWS_CA_BUNDLE
        value: /certs/ca.crt
          
      - name: REQUESTS_CA_BUNDLE
        value: /certs/ca.crt

      - name: VLLM_ENABLE_CUDA_COMPATIBILITY
        value: "1"

      - name: HF_HUB_OFFLINE
        value: "1"

      - name: AWS_ACCESS_KEY_ID
        value: <MY_ACCESS_KEY>
      
      - name: AWS_SECRET_ACCES_KEY
        value: <MY_SECRET_KEY>

      - name: AWS_ENDPOINT_URL
        value: "https://s3storage.abcd.hijk:9021"

      - name: RUNAI_STREAMER_LOG_TO_STDER
        value: "1"

      - name: RUNAI_STREAMER_LOG_LEVEL
        value: "DEBUG"

      - name: AWS_EC2_METADATA_DISABLED
        value: "true"

      - name: RUNAI_STREAMER_S3_USE_VIRTUAL_ADDRESSING
        value: "0"

the current runai version in the vllm pod deployed:

runai-model-streamer 0.16.0
runai-model-streamer-azure 0.16.0
runai-model-streamer-gcs 0.16.0
runai-model-streamer-s3 0.16.0

the current boto3 version in the vllm pod deployed:

boto3 1.43.29
botocore 1.43.29

the minio configs using mc alias ls

my-alias-name
        URL: https://s3storage.abcd.hijk:9021
        AccessKey: <MY_ACCESS_KEY>
        SecretKey: <MY_SECRET_KEY>
        API: s3v4
        Path: auto
        Src: /home/...../.mc/config.json

[noa-neria]

@agkjwork there are two typos in the environment variables:
AWS_SECRET_ACCES_KEY should be AWS_SECRET_ACCESS_KEY (missing S)
RUNAI_STREAMER_LOG_TO_STDER should be RUNAI_STREAMER_LOG_TO_STDERR (missing R)
However, that does not seem to be the problem since list_safetensors and pull_files worked correctly. The issue happened inside libstreamer.so

Let's check first if the issue was already fixed in #150 (will be released in the next version)
May you please try to install the streamer artifacts built in actions for #150 instead of 0.16.0 ?

However, I failed to reproduce your issue with 0.16.0 and TLS worked correctly for Minio.

The next step is to enable the AWS trace logs with RUNAI_STREAMER_S3_TRACE=1
Logs are written to a separate file in the same directory where the application is running

[agkjwork]

@agkjwork there are two typos in the environment variables: AWS_SECRET_ACCES_KEY should be AWS_SECRET_ACCESS_KEY (missing S) RUNAI_STREAMER_LOG_TO_STDER should be RUNAI_STREAMER_LOG_TO_STDERR (missing R) However, that does not seem to be the problem since list_safetensors and pull_files worked correctly. The issue happened inside libstreamer.so

Let's check first if the issue was already fixed in #150 (will be released in the next version) May you please try to install the streamer artifacts built in actions for #150 instead of 0.16.0 ?

However, I failed to reproduce your issue with 0.16.0 and TLS worked correctly for Minio.

The next step is to enable the AWS trace logs with RUNAI_STREAMER_S3_TRACE=1 Logs are written to a separate file in the same directory where the application is running

Hello, thanks for the update. The envs are correct in my helm chart, just made a couple of typos here as i could not copy and paste the content here but nonetheless, the issue still persists.

Would you be able to show me the setup and configs that you were using?

[noa-neria]

@agkjwork My setup was simple using AWS_CA_BUNDLE, AWS_ENDPOINT_URL=https://127.0.0.1:9443 and static AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID, with Minio and vLLM on the same host.
Please enable RUNAI_STREAMER_S3_TRACE=1 and provide the specific error messages.

[agkjwork]

@agkjwork My setup was simple using AWS_CA_BUNDLE, AWS_ENDPOINT_URL=https://127.0.0.1:9443 and static AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID, with Minio and vLLM on the same host. Please enable RUNAI_STREAMER_S3_TRACE=1 and provide the specific error messages.

Hello, i cant paste the full logs here but i got:

Creating client endpoint https://s3storage.abcd.hijk:9021
s3 request timeout is set to 1000ms
Setting s3 configuration userVirtualAddressing to 0
Setting s3 configuration ca certificate file to /certs/ca.crt
Using default authentication
created responder for 1 running requests [Type: N5runai3llm....
Responder incremented, new running count: 2
Waitng for responses

Failed to download s3 object of request 1 : socket is closed. ( aws-c-io: AWS_TO_SOCKET_CLOSED)
Error response code: File access error while waiting for response
Error response code: File access error while reading batches
Failed to read from file s3://my-bucket/models/google/gemma-4-31B-it/model-00001-of-00002.safetensors : error: response code: File access error

What envs are there to increase the request window? It seems to be able to hit my endpoint and retrieve a tensor file but the runai docs does not seem to have a timeout env that can be set.

On a side note, im running on a multi-gpu setup using nvlink, are there any additional envs that runai streamer needs to facilitate this?

[noa-neria]

@agkjwork These logs are from the streamer code. You need to check the logs of the aws cpp sdk itself.
Please enable the AWS trace logs with RUNAI_STREAMER_S3_TRACE=1
Important: Logs are written to a separate file in the same directory where the application is running. The aws trace logs are not in the streamer logs
Look for a file name format aws_sdk_<date>.log

Regarding the timeout, it is not the problem. You should analyze the sdk logs to understand why the connection is closed. Could be tls negotiation for example

[agkjwork]

I think i may found the root cause. I had assumed i was testing against minio, but I was actually hitting dell powerscale native S3 endpoint which was provisioned which constantly failed. Once I switched to a minio instance running powerscale underneath, the issue no longer occurred.

However, i am now encountering a new error with streaming.

Failed to download s3 object of request 5: Http connection channel shut down due to failure to meet throughput minimum ( aws-c-http: AWS_ERROR_HTTP_CHANNEL_THROUGHPUT_FAILURE)
Error response code: File access error while waiting for response
Error response code: File access error while reading batches
Failed to read from file s3://models/gemma-4-31B-it/model-00001-of-00002.safetensors ; error: response code: File access error
Releasing s3 client
Loading safetensors using Runai Model Streamer 0% Completed .......
Overall time to stream 58.3 GiB of all files to cpu: 14.73s, 4.0 GiB/s
Streamer shutting down
Stopping all s3 clients
Stopping responder...
Responder stopped and semaphore posted
responder stopped while waiting or after acquiring lock 
FinishedError while waiting for responses
Relasing s3 client
respnder stopped while waiting or after acquiring lock
responder stopped while waiting or after acquiring lock 
FinishedError while waiting for responses
FinishedError while waiying for responses
Releasing s3 client
Releasing s3 client
Responder shutdown
Shutting diwb s3 ckuebt wraooer 
Releasing all s3 clients
Failed to download s3 object of request 393: Http connection channel shut down de to meet throughput minimum ( aws-c-http: AWS_ERROR_HTTP_CHANNEL_THROUGHPUT_FAILURE)
Responder stopped, ignoring pushed reponse 
Failed to download s3 object of request 111: Http connection channel shut down de to meet throughput minimum ( aws-c-http: AWS_ERROR_HTTP_CHANNEL_THROUGHPUT_FAILURE)
Responder stopped, ignoring pushed reponse 
Failed to download s3 object of request 253: Http connection channel shut down de to meet throughput minimum ( aws-c-http: AWS_ERROR_HTTP_CHANNEL_THROUGHPUT_FAILURE)
Responder stopped, ignoring pushed reponse 

Printing of the "Failed to download s3 object of request " constantly repeats itself and then eventually it hit a EngineCore failed to start and it breaks.

My env:

- name: RUNAI_STREAMER_CONCURRENCY
  value: 4

If i were to set the concurrency value to 3 or even lower, sometimes the streamer actually works and deploy and sometimes the issue above happens.

Im not sure what part of the aws logs would help in this problem though.

Also, once the model is loaded, where are the tensor files cached? Ive looked through the directory of the pod and i dont see any models being cached. For hugging face we could append a "HF_HUB_OFFLINE" flag and the model will look for the cache directory, how does caching work for runai streamer? I just want to run runai streamer once once and if the vllm pod restarts, it will just pull from the cache instead if that is possible?

[noa-neria]

Regarding AWS_ERROR_HTTP_CHANNEL_THROUGHPUT_FAILURE - this may be due to throttling in your object storage backend. Try to find the configured limit. The streamer maximal bandwidth consists of the number of clients RUNAI_STREAMER_CONCURRENCY and the maximal bandwidth per client RUNAI_STREAMER_S3_TARGET_GBPS which is the target gigabits (default is 10 gbps which is 1.5 Gigabytes per second). Try to set RUNAI_STREAMER_CONCURRENCY=1 and small value for RUNAI_STREAMER_S3_TARGET_GBPS.
To investigate the source of your problem it will be helpful to look for errors in the aws sdk logs and consult with the docs and maintainers of dell powerscale native S3. I am only assuming it is throttling.

The streamer is reading the tensors from object storage directly to vLLM memory. Regarding caching files in local disk in addition to streaming, this feature #152 is under development.

[agkjwork]

Hello, I know this is a bit counterintuitive, but is there any way to throttle the streamer to stream below a certain throughput? For example, I'd like to cap it at 500 MiB/s or 600 MiB/s etc. Are there any envs to support this to enforce a bandwidth limit using runai streamer?

[noa-neria]

Yes there is. For S3:
RUNAI_STREAMER_CONCURRENCY=1 RUNAI_STREAMER_S3_TARGET_GBPS=4

GBPS is gigabit per second, so 4 is 500 mb