Skip to content

deployment.md

Latest commit

 

History

History
101 lines (79 loc) · 4.77 KB

deployment.md

File metadata and controls

101 lines (79 loc) · 4.77 KB
Raw

Step 1: Prerequisites to Deploy Granite-3.2-2b-Instruct Model on Xeon with Keycloak

Ensure the Enterprise Inference stack with Keycloak is already deployed before proceeding.

Edit core/scripts/generate-token.sh and set your values before sourcing it:

Variable Description
BASE_URL Hostname of your cluster (e.g. api.example.com), without https://
KEYCLOAK_ADMIN_USERNAME Keycloak admin username
KEYCLOAK_PASSWORD Keycloak admin password
KEYCLOAK_CLIENT_ID Keycloak client ID configured during EI deployment

Then run:

export HUGGING_FACE_HUB_TOKEN="your_token_here"

cd ~/Enterprise-Inference
source core/scripts/generate-token.sh

This exports: BASE_URL, KEYCLOAK_CLIENT_ID, KEYCLOAK_CLIENT_SECRET, and TOKEN.

Step 2: Deploy Granite-3.2-2b-Instruct Model

helm install vllm-granite-3-2-instruct ./core/helm-charts/vllm \
  --values ./core/helm-charts/vllm/xeon-values.yaml \
  --set LLM_MODEL_ID="ibm-granite/granite-3.2-2b-instruct" \
  --set global.HUGGINGFACEHUB_API_TOKEN="$HUGGING_FACE_HUB_TOKEN" \
  --set ingress.enabled=true \
  --set ingress.secretname="${BASE_URL}" \
  --set ingress.host="${BASE_URL}" \
  --set oidc.client_id="$KEYCLOAK_CLIENT_ID" \
  --set oidc.client_secret="$KEYCLOAK_CLIENT_SECRET" \
  --set apisix.enabled=true \
  --set tensor_parallel_size="1" \
  --set pipeline_parallel_size="1"

Step 3: Verify the Deployment

kubectl get pods
kubectl get apisixroutes

Expected Output:

NAME                                          READY   STATUS    RESTARTS
keycloak-0                                    1/1     Running   0
keycloak-postgresql-0                         1/1     Running   0
vllm-granite-3-2-instruct-<hash>-<hash>       1/1     Running   0

Note: The pod name suffix <hash>-<hash> is auto-generated by Kubernetes and will differ on each deployment. Ensure all pods show 1/1 Running.

NAME                                    HOSTS
vllm-granite-3-2-instruct-apisixroute   api.example.com

Step 4: Test the Deployed Model

curl -k https://${BASE_URL}/granite-3.2-2b-instruct-vllmcpu/v1/completions \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "ibm-granite/granite-3.2-2b-instruct",
    "prompt": "What is Deep Learning?",
    "max_tokens": 25,
    "temperature": 0
  }'

If successful, the model will return a completion response.

To undeploy the model

helm uninstall vllm-granite-3-2-instruct

Parameters

Parameter Description
--set LLM_MODEL_ID="ibm-granite/granite-3.2-2b-instruct" Defines the target model from Hugging Face to deploy.
--set global.HUGGINGFACEHUB_API_TOKEN="..." Authenticates access to gated or private Hugging Face models. Replace with your own secure token.
--set ingress.enabled=true Enables Kubernetes Ingress to expose the model service externally.
--set ingress.host="${BASE_URL}" Public hostname or FQDN for the inference endpoint (maps to your Ingress controller IP).
--set ingress.secretname="${BASE_URL}" Kubernetes TLS Secret used for HTTPS termination at the ingress layer.
--set oidc.client_id="..." Keycloak OIDC client ID used for token-based authentication.
--set oidc.client_secret="..." Keycloak OIDC client secret corresponding to the client ID.
--set apisix.enabled=true Enables APISIX as the API gateway for routing and authentication.
--set tensor_parallel_size="1" Number of tensor parallel workers. Set to the number of available CPUs/GPUs per node.
--set pipeline_parallel_size="1" Number of pipeline parallel stages. Typically 1 for single-node deployments.