Skip to content

phase-10-nim-selfhost.md

Latest commit

 

History

History
34 lines (20 loc) · 1.39 KB

phase-10-nim-selfhost.md

File metadata and controls

34 lines (20 loc) · 1.39 KB
Raw

Phase 10 — Self-host one NIM container

Goal: run a NIM container on a rented GPU VM and swap the app over to it. Proves the application layer does not care whether the endpoint is hosted or self-hosted — only the deploy does.

Prerequisites

This phase requires either a Linux machine with an NVIDIA GPU (A100/H100) or a cloud GPU rental. Cheapest path: 4 hours of Lambda Labs H100 PCIe (~$10).

Step 1 — Pull and run the NIM container

On the rented VM:

docker login nvcr.io   # username: $oauthtoken, password: NGC_API_KEY
docker run --rm --gpus all -p 8000:8000 \
  -e NGC_API_KEY=$NGC_API_KEY \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

Step 2 — Point the app at the self-hosted endpoint

Once the container is serving, change NVIDIA_BASE_URL in your local .env to the VM's IP and CHAT_MODEL to meta/llama-3.1-8b-instruct. Everything else just works because the API surface is identical.

Checkpoint

Send a chat query through the app with the updated .env. The response should come from the self-hosted NIM container on the rented VM. Verify by checking the VM's container logs for incoming requests.

What this demonstrates

The OpenAI-compatibility of NIM means hosted vs self-hosted is a deploy concern, not an application concern. The same client code works against both.

Next: Phase 11 — PersonaPlex voice interface