Suomea — Live AI Tutor

An interactive educational web application for learning Finnish through conversational practice using real-time speech recognition and TTS (Google Gemini Live). The project uses a small WebSocket middleware backend to host the Gemini live sessions securely so API keys remain on the server.

---

🚀 Overview

• Client: React + TypeScript (Vite). Captures audio (AudioWorklet) and communicates with the backend via WSS /live.
• Backend: Node + Express + ws running on Render (or your host). Manages Google Gemini Live sessions using @google/genai, forwards audio & transcripts between client and Gemini, and handles tool calls (e.g., reportProgress).
• AI: Google Gemini Live — streaming transcription, TTS, and function/tool calls.
• Persistence & Auth: Supabase (Postgres, Auth, Storage).
• Payments: Stripe (via backend or Supabase integration).

---

📁 Key Files & Locations

• server/index.ts — WebSocket middleware & Gemini session logic.
• src/services/liveService.ts — Client-side WebSocket client and audio capture logic.
• src/components/LiveDialoguePage.tsx — UI that integrates LiveService.
• src/utils/audioUtils.ts — PCM encode/decode helpers.
• diagram.png — Architecture diagram (embedded above).

---

🔧 Environment Variables

Server (create server/.env, do not commit):

• GEMINI_API_KEY — Gemini API key (rotate immediately if leaked)
• PORT (optional, default: 4000)

Client (Vite env — create .env in project root):

• VITE_LIVE_SERVER_URL — wss://yourdomain/live (or ws://localhost:4000/live for local dev)
• VITE_SUPABASE_URL
• VITE_SUPABASE_ANON_KEY

Security: Never commit secrets. Use server/.env.example and add server/.env to .gitignore.

---

🧰 Local Development

1. Server

   • cd server
   • npm install
   • create server/.env with GEMINI_API_KEY (and PORT if needed)
   • npm run dev

2. Client

   • at repo root: npm install
   • set VITE_LIVE_SERVER_URL in root .env (or rely on fallback)
   • npm run dev

3. Open the app in your browser and click the mic to start a Live session. Watch server logs for GenAI session lifecycle logs.

Quick validation: wscat -c ws://localhost:4000/live then {"type":"init","prompt":"test"}.

---

🔁 Audio & WebSocket Behavior (summary)

• Client can send either:
  • Binary frames (ArrayBuffer) — efficient; server base64-encodes and forwards to Gemini, or
  • JSON frames { type: 'audio', data: '<base64>' }.
• Server calls session.sendRealtimeInput(...) to forward audio to Gemini.
• Gemini returns base64 audio and transcripts; server forwards as { type: 'audio' } and { type: 'transcript' }.
• Tool calls from Gemini (e.g., reportProgress) are forwarded to the UI as { type: 'stage_complete' }.

---

🔐 Security & Ops

• Keep GEMINI_API_KEY on the server only.
• Rotate keys immediately if accidentally committed.
• Add authentication to WS init (e.g., Supabase JWT) in production.
• Add rate-limiting and per-session timeouts to control costs.
• Buffer audio on server until the Gemini session is ready to avoid dropped frames.

---

📡 Deployment Notes (Render)

• Deploy server/ as a Node service on Render (set env vars via dashboard).
• Deploy the client separately (Vercel/Netlify/Render static site) and set VITE_LIVE_SERVER_URL to your server WSS endpoint.
• Ensure TLS and use wss:// for secure websocket traffic from HTTPS pages.

---

🧾 Database & Migrations

• Database migrations are in supabase/migrations/.
• Sample seed data in seed-data.sql.

---

🛠 Troubleshooting

• If WS closes immediately (close code 1005): check WS scheme (wss:// vs ws://), DevTools WS frames, and server logs for socket errors.
• If TypeError: session.sendRealtimeInput is not a function: ensure server awaits ai.live.connect() and buffers audio until the session is ready.
• If no messages from Gemini: verify ai.live.connect resolves and callbacks (onopen, onmessage, onerror) are firing in server logs.

---

⚠️ Production Readiness

This project is not production-ready and is provided as a reference implementation or starting point. Several critical backend security and reliability improvements are still needed before deploying to production.

🛠️ TODO — Backend Security & Enhancements

The following enhancements should be implemented before using this in production:

• WebSocket Authentication: The /live endpoint currently accepts connections without rigorous auth checks. Add proper JWT or session validation on every connection and message.
• Rate Limiting & Abuse Prevention: Implement per-IP, per-user, and per-session rate limits to prevent abuse and runaway API costs.
• Input Validation & Sanitization: Validate all inbound messages and media streams to prevent injection attacks and malformed payloads.
• Secrets Management: Rotate all exposed API keys (Gemini, HeyGen, Supabase, Stripe, FTP) and move them to a secrets manager (e.g., Render env vars, Vault, AWS Secrets Manager) instead of .env files.
• Cost Controls: Add per-session timeouts, token budgets, and audio duration limits to control AI API spend.
• TLS & CORS: Enforce wss:// in production and restrict CORS to trusted origins only.
• Logging & Monitoring: Add structured logging, request tracing, and alerting for session failures and anomalous traffic.
• Error Handling & Resilience: Implement retries, circuit breakers, and graceful degradation when third-party APIs (Gemini, HeyGen) are unavailable.
• Deployment Hardening: Run the backend behind a reverse proxy, disable unnecessary endpoints, and perform regular dependency audits.

---

🤝 Contributing

• Fork, implement feature/fix, open a PR with tests and manual verification steps.
• When modifying WS / Gemini logic, run a live session to validate behavior.

---

📄 License

This project is licensed under the MIT License. See LICENSE for details.