GPT

A from-scratch implementation of a GPT-style (Generative Pre-trained Transformer) language model, featuring a FastAPI backend for text generation and a React + Vite frontend. Implements bigram, single-head attention, and multi-head attention variants of the transformer architecture, trained on a custom text corpus.

---

🔗 Links

• Demo: http://localhost:8010/generate (local backend)
• Docs: CLAUDE.md — project architecture & data flow
• Issues: GitHub Issues
• Contributing: CONTRIBUTING.md

---

📑 Table of Contents

• Installation
• Usage
• Features
• Project Structure
• Tech Stack
• Dependencies & Packages
• Contributing
• License
• Security
• Code of Conduct

---

🚀 Installation

Prerequisites

• Python 3.10+
• Node.js 20+
• npm (comes with Node.js)
• CUDA-capable GPU (optional, for faster training/inference)

Backend Setup

cd backend
python -m venv venv

# Windows
venv\Scripts\activate

# macOS/Linux
source venv/bin/activate

pip install torch fastapi uvicorn numpy pydantic starlette

Frontend Setup

npm install

---

⚡ Usage

Run the Backend (FastAPI + Uvicorn)

cd backend
venv\Scripts\activate   # Windows
python main.py          # starts on http://localhost:8010 with reload

Generate text:

curl "http://localhost:8010/generate?tokens=100"

Or open http://localhost:8010/generate in a browser.

Run the Frontend (Vite + React)

npm run dev        # starts on http://localhost:5173
npm run build      # production build
npm run preview    # serve production build

Train the Model

cd backend
python training/train.py

Trains the multi-head attention model for 2,500 steps with AdamW (lr=3e-4, batch_size=64) and saves the checkpoint to backend/model/GPT.pt.

Run Standalone Inference

cd backend
python run.py

Loads model/GPT.pt and prints 1,000 generated tokens to stdout.

---

✨ Features

• Three model variants: vanilla bigram, single-head self-attention, multi-head self-attention
• Causal masking for autoregressive generation
• Character-level tokenization with dynamic vocabulary built from the dataset
• Position embeddings for attention-based variants
• FastAPI endpoint (/generate) with CORS enabled
• CUDA acceleration with automatic CPU fallback
• Configurable context window (256 tokens) and generation temperature
• React + Vite frontend ready for integration

---

📂 Project Structure

GPT/
├── backend/
│   ├── attention/
│   │   ├── multi_head/
│   │   │   └── Multihead_attention.py   # Multi-head self-attention model
│   │   └── single_head/
│   │       └── biModel.py               # Single-head self-attention model
│   ├── config/
│   │   ├── preprocessing.py             # Text cleaning utility
│   │   └── tokenizer.py                 # Character-level encoder/decoder + batching
│   ├── dataset/
│   │   └── input.txt                    # Training corpus (~1.1 MB)
│   ├── model/
│   │   └── GPT.pt                       # Trained model checkpoint
│   ├── module/
│   │   └── bigramModel.py               # Vanilla bigram (no attention)
│   ├── training/
│   │   └── train.py                     # Training loop (AdamW, 2500 steps)
│   ├── main.py                          # FastAPI app with /generate endpoint
│   ├── run.py                           # Standalone inference script
│   └── venv/                            # Python virtual environment
├── src/                                 # React frontend (Vite boilerplate)
│   ├── App.tsx
│   ├── main.tsx
│   └── assets/
├── public/                              # Static frontend assets
├── index.html                           # Vite entry point
├── package.json
├── vite.config.ts
├── tsconfig.json
├── tsconfig.app.json
├── tsconfig.node.json
├── CLAUDE.md                            # Project architecture documentation
└── README.md

---

🛠 Tech Stack

Languages

Frameworks & Libraries

Backend / API

DevOps / CI / Tools

---

📦 Dependencies & Packages

Runtime Dependencies (Python)

Click to expand

Package	Version	Description
	2.12.1	Deep learning framework
	0.138.1	Modern web framework for APIs
	0.49.0	ASGI server
	2.4.6	Numerical computing
	2.13.4	Data validation
	1.3.1	ASGI framework (FastAPI dependency)
	3.1.6	Templating engine
	0.7.0	Type annotation utilities

Runtime Dependencies (Node.js)

Click to expand

Package	Version	Description
	^19.2.7	UI library
	^19.2.7	React DOM renderer

Dev / Build / Test Dependencies (Node.js)

Click to expand

Package	Version	Description
	^8.1.0	Build tool & dev server
	~6.0.2	Type-safe JavaScript
	^6.0.2	Vite React plugin
	^1.69.0	Linter
	^19.2.17	React type definitions
	^19.2.3	React DOM type definitions
	^24.13.2	Node.js type definitions

---

🤝 Contributing

We welcome contributions! Please read our Contributing Guide for details on how to submit pull requests, report issues, and follow our code of conduct.

---

📜 License

This project is licensed under the MIT License — see the LICENSE file for details.

---

🛡 Security

Please review our Security Policy for information on reporting vulnerabilities and our responsible disclosure process.

---

📏 Code of Conduct

This project adheres to the Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code.

---

Made with ❤ by H0NEYP0T-466