v1.0 — Visual RAG · Voice · Video · Capstone

S3 Multimodal Lab v1.0 — first stable release

A hands-on lab for modern multimodal AI: 12 notebooks organized in 4 phases that build a production-grade Visual RAG / voice / video stack end-to-end.

What's inside

Phase 1 — Visual RAG basics (nb01–04)

• nb01 How VLMs see images — ViT patching + GPT-4o token math
• nb02 CLIP text→image search
• nb03 ColPali Visual RAG — late interaction over PDF pages, no OCR
• nb04 Grounded generation — strict schema + visual_evidence + confidence gate

Phase 2 — Multilingual (nb05)

• nb05 ColPali vs Tesseract+OCR head-to-head on Chinese financial PDFs

Phase 3 — Voice (nb06–08, nb11)

• nb06 faster-whisper basics (multilingual, word timestamps)
• nb07 ZH-EN code-switching deep dive — 5-sample eval, two metrics (CER + English-word recall), three measured mitigation strategies
• nb08 Voice agent — sync + streaming sequential pipeline with per-stage latency budget
• nb11 Speech-to-speech (GPT-4o Realtime) head-to-head vs nb08

Phase 4 — Video (nb09–10)

• nb09 Dual-index (CLIP frames + Whisper segments) with RRF fusion over YouTube / B站 lectures
• nb10 Gemini long-context native video vs DIY pipeline

Capstone (nb12)

• Voice in → Whisper → ColPali → grounded GPT-4o → confidence gate → TTS. One async CapstoneAgent class wiring everything together.

Engineering quality

• s3lab/ package with canonical pure-Python utilities (cost math, MaxSim, RRF, normalization)
• 95 pytest sanity assertions running in CI (~110ms), no API keys / GPUs needed
• Per-phase eval harness in evals/ with ground truth + pass/fail thresholds — Phase 3 (code-switching) fully self-contained via TTS-generated audio
• Every notebook validated: parseable JSON, starts with H1, no committed outputs
• Cross-platform device autodetection (CUDA → MPS → CPU)

Designed for

• Recordable video series — each commit is a chapter, each notebook ~15–25 min explanation
• Students cloning to follow along — git pull after each chapter
• Engineers wiring multimodal into production — every defensive design choice maps back to its motivating S3 deck section

Quick start

git clone https://github.com/zyziyun/s3-multimodal-lab
cd s3-multimodal-lab
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
brew install poppler tesseract tesseract-lang ffmpeg
cp .env.example .env  # add OPENAI_API_KEY
jupyter lab

Acknowledgments

Builds on top of: ColPali (Faysse et al., 2024) · LLaVA · faster-whisper · CLIP · OpenAI GPT-4o · Google Gemini · the wider open VLM community.