Skip to content

03-examples.md

Latest commit

 

History

History
215 lines (156 loc) · 5.24 KB

03-examples.md

File metadata and controls

215 lines (156 loc) · 5.24 KB
Raw

Examples

How to connect to inference-hub from different languages, tools, and frameworks.

For all examples below, replace:

  • SERVER_IP with the machine's IP address (or localhost if on the same machine)
  • YOUR_API_KEY with your actual API key (from ./hub add-key)

Python

Quick test

pip install openai
export INFERENCE_HUB_URL="http://SERVER_IP:4200/v1"
export INFERENCE_HUB_KEY="YOUR_API_KEY"
python examples/test_connection.py

This lists available models, sends a test message, and confirms the connection works.

Interactive chat

python examples/chat_session.py

A multi-turn chat in your terminal. Maintains conversation history.

Streaming

python examples/streaming.py

Watch tokens arrive in real time.

Image input (vision)

python examples/multimodal.py --file path/to/photo.jpg
python examples/multimodal.py --file https://example.com/photo.jpg --prompt "What breed is this dog?"
python examples/multimodal.py --file diagram.png --model large

Sends an image (local file or URL) alongside a text prompt. All default Gemma 4 models accept images — small (E4B) and large (26B MoE).

Audio input

python examples/audio.py --file clip.wav
python examples/audio.py --file speech.mp3 --prompt "Translate this to French."

Audio requires a Gemma 4 E2B or E4B model. The default small model (Gemma 4 E4B) supports audio out of the box. The large model (26B MoE) is text + image only and will return a 400 on audio input.

In your own code

from openai import OpenAI

client = OpenAI(
    base_url="http://SERVER_IP:4200/v1",
    api_key="YOUR_API_KEY",
)

response = client.chat.completions.create(
    model="small",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain transformers in one paragraph."},
    ],
)
print(response.choices[0].message.content)

curl

Simple request

curl http://SERVER_IP:4200/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "small",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

List models

curl http://SERVER_IP:4200/v1/models \
  -H "Authorization: Bearer YOUR_API_KEY"

Streaming

curl http://SERVER_IP:4200/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "model": "small",
    "stream": true,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

JavaScript / TypeScript

npm install openai
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://SERVER_IP:4200/v1",
  apiKey: "YOUR_API_KEY",
});

const response = await client.chat.completions.create({
  model: "small",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(response.choices[0].message.content);

Open WebUI

Open WebUI gives you a ChatGPT-like browser interface for your models.

docker run -d -p 3000:8080 \
  -e OPENAI_API_BASE_URL=http://HOST_IP:4200/v1 \
  -e OPENAI_API_KEY=YOUR_API_KEY \
  --add-host=host.docker.internal:host-gateway \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

Replace HOST_IP with the server's LAN IP (not localhost — the container needs to reach the host network). Then open http://localhost:3000 in your browser and select the small or large model.

Agent Frameworks

Inference-hub works as a private backend for AI agent frameworks. Both OpenAI and Anthropic API patterns are supported.

OpenAI-compatible agents

Any framework using the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://SERVER_IP:4200/v1",
    api_key="YOUR_API_KEY",
)

response = client.chat.completions.create(
    model="small",
    messages=[{"role": "user", "content": "Hello"}],
)

Anthropic-compatible agents

Any framework using the Anthropic SDK:

import anthropic

client = anthropic.Anthropic(
    base_url="http://SERVER_IP:4200/anthropic",
    api_key="YOUR_API_KEY",
)

message = client.messages.create(
    model="small",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)
print(message.content[0].text)

Environment variable pattern

Most frameworks accept endpoint configuration via environment variables:

# For OpenAI-compatible frameworks
export OPENAI_BASE_URL="http://SERVER_IP:4200/v1"
export OPENAI_API_KEY="YOUR_API_KEY"

# For Anthropic-compatible frameworks
export ANTHROPIC_BASE_URL="http://SERVER_IP:4200/anthropic"
export ANTHROPIC_API_KEY="YOUR_API_KEY"

Both endpoints serve the same models. Use whichever matches your framework's SDK.

Any OpenAI- or Anthropic-compatible tool

The pattern is always the same:

Setting OpenAI SDK Anthropic SDK
Base URL / API Base http://SERVER_IP:4200/v1 http://SERVER_IP:4200/anthropic/v1
API Key YOUR_API_KEY YOUR_API_KEY
Model name small or large small or large

This includes LangChain, LlamaIndex, AutoGen, CrewAI, Semantic Kernel, Continue.dev, Cursor, Aider, and anything else with an OpenAI- or Anthropic-compatible mode.