Beacon

Beacon is a LLM gateway/proxy service that routes OpenAI compatible API requests.

Quick overview

• Gateway (server) accepts OpenAI-compatible requests on /v1/* endpoints and routes them to connected nodes.
• Nodes connect to the gateway over Socket.io (useful for complex NAT situations) and authenticate using a node token created by the included CLI.
• A MongoDB instance stores users, nodes, API keys, and model metadata.

When a request comes in, the gateway checks whether the requested model is registered. If a connected node is available for that model, Beacon routes the request to that node. If no eligible node is available and the model is configured with cloud fallback, Beacon forwards the request to an OpenAI API-compatible upstream service.

flowchart LR
    Client[Client / App] --> Gateway[Beacon Gateway]

    Gateway --> Registry[Model Registry]
    Gateway --> Nodes[Connected GPU Nodes]

    Nodes --> GPU1[Consumer GPU]
    Nodes --> GPU2[Datacenter GPU]

    Gateway --> Fallback[OpenAI-Compatible Cloud Provider]

Loading

Why Beacon?

Modern GPU capacity is often fragmented. A developer may have a gaming GPU sitting idle. A lab may have datacenter GPUs that are only used during business hours. A team may want local inference when possible, but still needs cloud fallback for availability.

Beacon aims to provide:

• A single OpenAI-compatible gateway for clients
• Dynamic routing to connected GPU nodes
• Model-aware request dispatch
• Optional fallback to OpenAI-compatible cloud providers
• A path toward cost-efficient distributed inference

Prerequisites

• Node.js 18+ and npm
• MongoDB (local or remote)
• If using local model providers, the node needs access to an OpenAI-compatible inference endpoint (e.g. Ollama).

Install

Clone the repo and build the Docker image (recommended):

git clone <repo-url>
cd beacon
docker build -t beacon:latest .

Pull the exiting image:

docker pull ghcr.io/domkalan/beacon:latest

(If you prefer running from source, see BUILDING.md for full build and dev instructions.)

Environment

The gateway and node use these environment variables (defaults listed):

• MONGO_SRV — MongoDB connection string (default: mongodb://localhost:27017)
• MONGO_DBNAME — Database name (default: beacon)
• BEACON_SERVER — Gateway URL for nodes (default: http://localhost:3000)
• BEACON_TOKEN — Node token (generated by CLI)
• OPENAI_BASE_URL — Node OpenAI-compatible base URL (default: http://localhost:11434/v1)
• OPENAI_API_KEY — Node provider API key (default: ollama when using Ollama)

Export them in your shell or use a process manager (example using exported vars is shown below).

Run the gateway (server)

Recommended (Docker):

Create a user network so gateway and node containers can address each other by name:

docker network create beacon-net || true
docker run -d --name beacon --network beacon-net -p 3000:3000 \
  -e MONGO_SRV="mongodb://host.docker.internal:27017" \
  -e MONGO_DBNAME="beacon" \
  beacon:latest

Quick single-host run (no custom network):

docker run -d --name beacon -p 3000:3000 \
  -e MONGO_SRV="mongodb://localhost:27017" \
  -e MONGO_DBNAME="beacon" \
  beacon:latest

The gateway listens on port 3000 by default.

For development or building from source, see BUILDING.md.

Using the included CLI

The repository includes a small CLI (src/cli.ts) for managing users, nodes, API keys, and models.

Recommended: run it with npx ts-node during development or inside a container as shown below.

1. Create an administrative user (replace email/password):

npx ts-node src/cli.ts users add admin@example.com YourPassword123

2. Create a node and get its token (associate it with the email you just created):

npx ts-node src/cli.ts nodes add my-node admin@example.com

The command prints a table containing the token value. Save that token — it is used by nodes to authenticate with the gateway.

3. (Optional) Create an API key for clients to call the gateway:

npx ts-node src/cli.ts keys add "client-key" admin@example.com

The command prints the API key value. Use that value as a Bearer token when calling the gateway:

curl http://localhost:3000/v1/chat/completions \
 -H "Authorization: Bearer <API_KEY_VALUE>" \
 -H "Content-Type: application/json" \
 -d '{ "model":"llama-3.1-8b","messages":[{"role":"user","content":"Hello"}] }'

Starting a Node

A node is a separate process that connects to the gateway over Socket.io and handles model inference requests.

Recommended (Docker):

Run a node container on the same beacon-net so it can reach the gateway by container name:

docker run -d --name beacon-node --network beacon-net \
  -e BEACON_TOKEN="<NODE_TOKEN_FROM_CLI>" \
  -e BEACON_SERVER="http://beacon:3000" \
  -e OPENAI_BASE_URL="http://<local-provider-host>:11434/v1" \
  -e OPENAI_API_KEY="ollama" \
  beacon:latest node node.js

Notes:

• The CLI nodes add command prints the node token value; use that as BEACON_TOKEN above.
• If you run gateway on the host and the node in Docker, point BEACON_SERVER at the host (for example http://host.docker.internal:3000 on supported platforms).
• For building or running from source (non-Docker), see BUILDING.md.

Notes:

• The node will attempt to identify its available models by requesting models.list from the remote provider (the node's OPENAI_BASE_URL).
• The gateway verifies node tokens against the nodes collection in MongoDB. Create nodes with the CLI first so the token exists server-side.

Troubleshooting

• If the gateway cannot connect to MongoDB, set MONGO_SRV to a reachable MongoDB URI and MONGO_DBNAME as needed.
• If nodes fail to authenticate, ensure the token you exported matches the token returned by nodes add and that the gateway is pointed at the same MongoDB instance.

Development tips

• Use npm run develop to run the gateway without building.
• Use npx ts-node for CLI and node during development.

Contributing

Contributions are welcome. Fork, create a branch, implement changes, and open a PR.

git checkout -b feature/my-feature

Roadmap / To-do

• Full OpenAI API compatibility
• Lock nodes when generating
• Cache model list to prevent heavy network load
• Implement streaming response support for Socket.io connected nodes
• Web UI and dashboard instead of CLI
• Token tracking per node, per model, per provider, per user

License

MIT