ModelRouter

Asynchronous HTTP API with streaming that orchestrates multiple LLM providers with automatic fallback and observability.

---

Main Features

• Multi-provider Orchestration: Automatic fallback between:
  • Groq
  • OpenRouter
  • OpenAI
  • Ollama
• Native Streaming: Support for Server-Sent Events (SSE).
• Resilience: Rate limiting, temporary blocklisting, and exponential backoff.
• Production Ready: Includes Prometheus metrics, structured logging, and Docker deployment.

---

Quick Start (Docker)

1. Configuration

git clone https://github.com/HC-ONLINE/ModelRouter.git
cd ModelRouter
cp .env.example .env

Edit the .env file with your API keys.

---

2. Deployment

docker-compose up --build

The API will be available at http://localhost:8000.

---

Basic Usage

Chat (Non-Streaming)

curl -X POST http://localhost:8000/chat \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}], "provider": "groq"}'

---

Streaming

curl -N -X POST http://localhost:8000/stream \
  -H "Authorization: Bearer your_api_key" \
  -d '{"messages": [{"role": "user", "content": "Tell a story"}]}'

---

Detailed Documentation

• Architecture - How it works internally.
• Configuration - Environment variables and rate limits.
• Usage Examples - Examples using curl and fetch.
• Development - Contribution guide, tests and linting.
• Observability - Metrics and logs.
• Security - Security and legal notes.

These documents are located in the docs/ folder.

---

Project Structure

ModelRouter/
├── api/                # Core logic (FastAPI, Router, Orchestrator)
├── api/providers/      # Adapters (Groq, OpenRouter, Ollama)
├── api/infra/          # HTTP clients and Redis integrations
├── tests/              # Test suite
├── docs/               # Technical documentation
└── docker-compose.yml

---

License

This project is licensed under the Apache-2.0 License. See LICENSE for details.

---

Legal Disclaimer

This project is intended for individual or self-hosted use. Make sure to:

• Read and comply with the Terms of Service of the providers you use
• Do not use provider rotation to evade usage limits
• Respect rate limits and policies of each provider
• Do not store/process sensitive data without appropriate security measures

The author assumes no liability for misuse of this software.

---

Community

Made with ❤️ by HC-ONLINE. See ROADMAP.md for next steps

⭐ If you find it useful, please give it a star on GitHub ⭐