LLM Evaluation Framework

A benchmarking system for evaluating and comparing outputs from multiple large language models. Runs prompts across models, scores responses using ROUGE, BLEU, and an LLM-as-judge approach, and displays results in a side-by-side comparison dashboard.

![Demo GIF here]

What it does

Submit a prompt and select which models to test. The system generates responses from each model, computes automatic metrics against a reference answer, and uses a second LLM call to score each response on factuality, coherence, and relevance. All results are stored and browsable in the history view.

Features

• Multi-model comparison: llama-3.1-8b-instant, llama-3.3-70b-versatile, mixtral-8x7b-32768, gemma2-9b-it
• ROUGE-1, ROUGE-2, ROUGE-L, and BLEU scoring against reference answers
• LLM-as-judge evaluation using llama-3.3-70b-versatile as the scoring model
• Three built-in evaluation datasets: factual QA, summarization, reasoning
• SQLite-backed run history with full result detail
• BYOK: bring your own Groq API key, never stored server-side
• Clean two-tab dashboard: evaluate and history views

Tech Stack

Component	Technology
Backend	FastAPI, Python 3.11
LLM Inference	Groq API (httpx, no SDK)
Text Metrics	rouge-score, NLTK BLEU
LLM Judge	llama-3.3-70b-versatile via Groq
Storage	SQLite
Frontend	HTML, CSS, JavaScript

Prerequisites

• Python 3.11+
• A free Groq API key from console.groq.com

Setup

git clone https://github.com/SaiVenkataGaneshBandaluppi/llm-evaluation-framework
cd llm-evaluation-framework
python -m venv venv
venv\Scripts\activate
pip install -r requirements.txt
python -c "import nltk; nltk.download('punkt'); nltk.download('punkt_tab')"

Run

uvicorn app.main:app --host 0.0.0.0 --port 8011

Open http://localhost:8011

Enter your Groq API key in the interface. Get a free key at console.groq.com.

Usage

1. Enter your Groq API key in the API Key field.
2. Type a prompt in the Prompt box.
3. Optionally provide a reference answer to enable ROUGE and BLEU scoring.
4. Check one or more models to test.
5. Click Analyze to run the evaluation.
6. View side-by-side results with metric scores and judge ratings.
7. Use the dataset quick-run section to benchmark against built-in datasets.
8. Switch to the History tab to browse past evaluations.

API

All endpoints except /api/health and /api/models require an X-API-Key: your-groq-key header.

POST /api/evaluate          - run evaluation on a custom prompt
POST /api/evaluate/dataset  - run evaluation on a built-in dataset
GET  /api/runs              - list recent evaluation runs
GET  /api/runs/{run_id}     - get full run detail
GET  /api/models            - list supported models
GET  /api/health            - health check

Example

curl -X POST http://localhost:8011/api/evaluate \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-groq-key" \
  -d '{"prompt":"What is the capital of France?","models":["llama-3.1-8b-instant"],"reference":"Paris"}'

Architecture

Prompt + Models -> Groq Inference -> ROUGE + BLEU Scoring -> LLM Judge -> SQLite -> Dashboard

License

MIT

Author

Sai Venkata Ganesh Bandaluppi