🤖 Voice-Controlled Robotic Arm — Whisper + LLaMA 3.1 8B + SmolVLA + LIBERO

Big Data Analytics Course Project — Gebze Technical University Fouad Aladhami · İsmail Eren Duman · Esra Tahinç

---

Table of Contents

1. 🎬 Demo
2. Project Overview
3. Quick Start — How to Run
4. Repository Structure
5. Pipeline Architecture
6. Stage 1 — Whisper Speech-to-Text
7. Stage 2 — Spark Cosine Similarity Task Matching
8. Stage 3 — LLaMA 3.1 8B JSON Parser
9. Stage 4 — SmolVLA + LIBERO Simulation
10. Fine-Tuning SmolVLA on JSON Instructions
11. Training Data Preparation
12. Evaluation Results
13. Modal Cloud Deployment
14. Web UI
15. Challenges & Design Decisions
16. References

---

🎬 Demo

Web UI — Pipeline in Action

Pipeline stages (Whisper → Cosine Match → SmolVLA) running in real-time with live console output

Live robot simulation video streaming from the A100 GPU while SmolVLA executes the task

Full Demo — Multiple LIBERO Tasks

---

1. Project Overview

This project bridges the gap between natural human speech and physical robotic manipulation. A user speaks a command such as "pick up the black bowl on the stove and place it on the plate"; the system transcribes it, semantically matches it to a LIBERO benchmark task, converts it to a structured JSON instruction via an LLM, and executes it with a fine-tuned Vision-Language-Action (VLA) policy — all streamed live to a browser UI.

Key novelty: SmolVLA is fine-tuned to understand JSON-encoded task descriptions (instead of plain English), creating a clean bridge from the LLM parser output directly to VLA inference.

Technology Stack

Component	Technology
Speech-to-Text	Whisper large-v3 (faster-whisper, int8_float16)
Task Matching	all-MiniLM-L6-v2 + Apache Spark cosine similarity
Instruction Parser	LLaMA 3.1 8B Instruct (via HuggingFace Transformers)
VLA Policy	SmolVLA fine-tuned on JSON instructions
Simulation	LIBERO benchmark (Franka Panda, MuJoCo/Robosuite)
Cloud GPU	Modal A100 (serverless, ~21 GB VRAM total)
Local Server	FastAPI + SSE streaming
Frontend	Vanilla HTML/CSS/JS

---

2. Quick Start — How to Run

Prerequisites

# Python 3.11+ required
git clone https://github.com/fouad1233/Speech_control_compatible_with_le_robot
cd Speech_control_compatible_with_le_robot
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Step 1 — Configure secrets

Create a .env file in whisper_smolvla_libero_pipeline/:

modal_read_access_token=hf_YOUR_HUGGINGFACE_TOKEN

Set up Modal authentication (one-time):

modal setup   # follow browser login

Step 2 — Deploy the Modal backend

cd whisper_smolvla_libero_pipeline
modal deploy unified_modal_app.py

This builds a Docker image on Modal with:

• Debian Slim + CUDA drivers
• LeRobot (with LIBERO extras) cloned from GitHub
• Whisper, SentenceTransformers, Transformers, HuggingFace Hub
• All local .py and .json files copied into /app

First deploy takes ~10–15 minutes (image build). Subsequent deploys are fast.

Step 3 — Run the pipeline

modal run unified_modal_app.py

Step 4 — Open the UI

Navigate to http://localhost:8000 in your browser.

1. Click Power On — waits ~2–3 minutes for the A100 to warm up (loads all 4 models)
2. Click the 🎤 microphone, speak your command, click again to stop
3. Watch the 4-stage pipeline execute live with real-time progress
4. The robot simulation video streams frame-by-frame to the browser

Alternative: Use the example chips (text shortcuts) to skip microphone recording.

3. Repository Structure

Speech_control_compatible_with_le_robot/
│
├── whisper_smolvla_libero_pipeline/     # Main production pipeline
│   ├── unified_modal_app.py             # The full Modal pipeline (all 4 stages)
│   ├── pipeline_server.py               # Local FastAPI server + SSE bridge
│   ├── index.html                       # Web UI (single file, no build step)
│   ├── libero_task_mapping.json         # LIBERO task descriptions (all suites)
│   └── libero_task_mapping_parsed_training.json  # JSON-parsed task instructions
│
├── smol-vla-libero-with-text-mapping/  # Fine-tuning & evaluation
│   ├── smolvla_finetune_parsed_json.py  # Full fine-tuning script (Kaggle)
│   ├── libero_task_mapper.py            # Task mapper utility
│   ├── libero_spark_matcher.py          # Standalone Spark matcher
│   ├── libero_task_mapping_parsed.json  # Full parsed task file (with confidence)
│   └── libero_task_mapping_parsed_training.json  # Stripped training file
│
├── llm_modal_setup/                    # LLM batch conversion tools
│   ├── batch_convert.py                # Batch LLaMA parsing of all LIBERO tasks
│   ├── prepare_training_data.py        # Strips inference-only fields for training
│   └── llm_inference.py               # Single-task LLM inference utility
│
├── SO100_Simulation/                   # Initial (deprecated) SO100 + PyBullet experiments
├── whisper_modal_setup/                # Standalone Whisper Modal deployment
└── midterm_report.tex                  # IEEE-format midterm report

---

4. Pipeline Architecture

🎤 Microphone / Text Input
         │
         ▼
┌─────────────────────────────┐
│  Stage 1: Whisper large-v3  │  ← faster-whisper, int8_float16, English locked
│  Speech → Text transcript   │
└──────────────┬──────────────┘
               │  "pick up the black bowl on the stove..."
               ▼
┌────────────────────────────────────────┐
│  Stage 2: Cosine Similarity Matching   │  ← all-MiniLM-L6-v2 + Spark
│  Transcript → Best LIBERO task         │  ← Suite filter (spatial/object/goal/10/90)
└──────────────────────┬─────────────────┘
                       │  suite=libero_spatial, task_id=3, score=0.912
                       ▼
┌─────────────────────────────────────────┐
│  Stage 3: LLaMA 3.1 8B JSON Parser     │  ← Few-shot prompting, bfloat16
│  Transcript → Compact JSON instruction  │
└───────────────────────┬─────────────────┘
                        │  {"steps":[{"step":1,"action":"pick up","object":"bowl",...}]}
                        ▼
┌──────────────────────────────────────────────────────┐
│  Stage 4: SmolVLA + LIBERO Simulation                │  ← A100 GPU, MuJoCo/Robosuite
│  JSON instruction → Robot action → Reward / Video    │  ← fouad1233/smolvla_libero_json_t4
└──────────────────────────────────────────────────────┘

All stages run inside a single Modal A100 container. The local server streams results back to the browser via Server-Sent Events (SSE).

---

5. Stage 1 — Whisper Speech-to-Text

Model: OpenAI Whisper large-v3 Library: faster-whisper (CTranslate2 backend) Precision: int8_float16 (mixed — fast on A100, ~3 GB VRAM)

self.whisper = WhisperModel("large-v3", device="cuda", compute_type="int8_float16")
segments, info = self.whisper.transcribe(audio_file, beam_size=5, language="en")
transcript = " ".join([segment.text for segment in segments]).strip()

Key decisions:

• language="en" is hardcoded — forces English decoding, reduces latency and hallucinations
• Audio is uploaded as audio/webm blob from the browser MediaRecorder API
• beam_size=5 balances speed vs. accuracy

The stage yields a transcribed event with the full transcript text, which is then consumed by both the matcher and the LLM parser.

---

6. Stage 2 — Spark Cosine Similarity Task Matching

Model: sentence-transformers/all-MiniLM-L6-v2 Framework: Apache Spark (for scalable similarity computation) Dataset: All LIBERO task descriptions from libero_task_mapping.json

How it works

On cold start, all LIBERO task descriptions are embedded once and stored as a tensor:

self.matcher = SentenceTransformer('all-MiniLM-L6-v2', device="cuda")
texts = [t["language_instruction"] for t in self.task_list]
self.task_embeddings = self.matcher.encode(texts, convert_to_tensor=True)

At inference time, the transcript is embedded and cosine similarity is computed against all tasks:

query_emb = self.matcher.encode(transcript, convert_to_tensor=True)
cos_scores = util.cos_sim(query_emb, self.task_embeddings)[0]

Suite Filtering

Users can select which LIBERO suites to search (spatial / object / goal / long / 90). Non-selected suites are excluded by forcing their scores to -1.0:

if allowed_suites:
    for i, t in enumerate(self.task_list):
        if t["suite"] not in allowed_suites:
            cos_scores[i] = -1.0

The top-5 matches are returned and visualized as a bar chart in the UI.

Available Suites

Suite	Tasks	Notes
libero_spatial	10	Spatial object placement tasks
libero_object	10	Object manipulation tasks
libero_goal	10	Goal-conditioned tasks
libero_10	10	Long-horizon tasks
libero_90	90	Diverse task set

---

7. Stage 3 — LLaMA 3.1 8B JSON Parser

Model: unsloth/Meta-Llama-3.1-8B-Instruct Precision: bfloat16, device_map="auto" VRAM: ~16 GB

Purpose

The VLA model was fine-tuned on JSON-encoded instructions (not plain English). The LLM parser converts the Whisper transcript into the exact same compact JSON format used during training, closing the loop between natural language and the VLA.

JSON Schema

{
  "steps": [
    {
      "step": 1,
      "action": "grab|move|place|push|rotate|lift|stop|pick up|open|close|turn on|turn off|stack|put",
      "object": "object name",
      "color": "red|blue|green|yellow|white|orange|black|unknown",
      "location": "location description"
    }
  ]
}

Note: confidence, ambiguous, and ambiguity_reason fields are stripped before feeding to the VLA — matching the exact training format.

Few-Shot Prompting

Three examples are injected into the chat context:

Input	Output
"pick up the black bowl on the stove and place it on the plate"	{"steps":[{"step":1,"action":"pick up","object":"bowl","color":"black","location":"stove"},{"step":2,"action":"place","object":"bowl","color":"black","location":"plate"}]}
"grab the red cube"	{"steps":[{"step":1,"action":"grab","object":"cube","color":"red","location":"unknown"}]}
"grab it"	Returns ambiguous: true with reason

Field Stripping (Training Alignment)

STEP_FIELDS_TO_REMOVE = {"confidence"}
PARSED_FIELDS_TO_REMOVE = {"ambiguous", "ambiguity_reason"}

# Strip, then compact-serialize — same as build_json_lookup() in training
compact = json.dumps(clean_parsed, separators=(",", ":"))

This ensures the JSON fed to SmolVLA at inference time is byte-for-byte identical to the format it was trained on.

---

8. Stage 4 — SmolVLA + LIBERO Simulation

Model: fouad1233/smolvla_libero_json_t4 (our fine-tuned checkpoint) Base: HuggingFaceVLA/smolvla_libero Environment: LIBERO (MuJoCo + Robosuite, Franka Panda robot) Display: Headless Xvfb virtual display on Modal

Execution Flow

# 1. Build environment from the matched task
env_cfg = make_env_config("libero", task=suite_name, task_ids=[task_id])
envs = make_env(env_cfg, n_envs=1, use_async_envs=False)

# 2. Load our fine-tuned policy
policy = make_policy(cfg=policy_cfg, env_cfg=env_cfg)
policy.eval()

# 3. Inference loop — up to 500 steps
for step in range(max_steps):
    obs["task"] = [compact_json]   # ← JSON instruction (not English!)
    action = policy.select_action(obs)
    observation, reward, terminated, truncated, _ = vec_env.step(action)
    if reward > 0:
        success = True
        break

Live Video Streaming

Each simulation step's camera frame is JPEG-encoded and Base64-streamed to the browser:

img = Image.fromarray(frame)
img = img.rotate(180)  # Robosuite renders upside down
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=70)
b64_img = base64.b64encode(buf.getvalue()).decode("utf-8")
yield {"type": "frame", "data": {"image_b64": b64_img, "step": step}}

The browser buffers all frames and replays them at 12 FPS after evaluation completes.

---

9. Fine-Tuning SmolVLA on JSON Instructions

Script: smol-vla-libero-with-text-mapping/smolvla_finetune_parsed_json.py Platform: Kaggle (T4 GPU, free tier) or Kaggle Pro (A100) Base model: HuggingFaceVLA/smolvla_libero (pretrained on LIBERO with English instructions)

Key Idea

The HuggingFaceVLA/libero dataset stores task descriptions as English text in meta/tasks.parquet. We patch this file to replace every English task description with its JSON equivalent before training starts.

Step-by-Step

1. Download the LIBERO dataset (~18 GB)

snapshot_download(repo_id="HuggingFaceVLA/libero", repo_type="dataset",
                  local_dir="/kaggle/tmp/libero_dataset_original")

2. Build a JSON lookup table

def build_json_lookup(parsed_tasks):
    lookup = {}
    for entry in parsed_tasks:
        key = entry["task_text"].strip().lower()
        compact = json.dumps(entry["parsed"], separators=(",", ":"))
        lookup[key] = compact
    return lookup

3. Patch meta/tasks.parquet — replace English text with JSON strings in the index

for original_text in df.index:
    key = str(original_text).strip().lower()
    if key in json_lookup:
        new_index.append(json_lookup[key])
df.index = new_index
df.to_parquet(tasks_pq, index=True)

4. Run lerobot-train

xvfb-run -a lerobot-train \
  --policy.path=HuggingFaceVLA/smolvla_libero \
  --dataset.root=/kaggle/tmp/libero_dataset_original \
  --output_dir=/kaggle/working/smolvla_finetune_output \
  --policy.repo_id=fouad1233/smolvla_libero_json_t4 \
  --policy.device=cuda \
  --policy.use_amp=true \
  --batch_size=4 \
  --steps=15000 \
  --save_freq=3000 \
  --policy.train_expert_only=false \
  --policy.freeze_vision_encoder=true \
  --wandb.enable=false

Training Configuration (T4 Profile)

Parameter	Value
Base checkpoint	HuggingFaceVLA/smolvla_libero
GPU	Kaggle T4 (16 GB VRAM)
Batch size	4
Steps	15,000 (~3 hours)
VLM backbone	Trained (unfrozen)
Vision encoder (SigLIP)	Frozen (saves VRAM)
AMP	Enabled (bfloat16)
Published to	fouad1233/smolvla_libero_json_t4

---

10. Training Data Preparation

Phase 1 — LLM Batch Parsing

Script: llm_modal_setup/batch_convert.py

All 130 LIBERO task descriptions were batch-processed through LLaMA 3.1 8B on Modal to generate libero_task_mapping_parsed.json. Each entry contains:

• Original English task text
• Full LLM output including confidence, ambiguous, ambiguity_reason

Phase 2 — Strip Inference-Only Fields

Script: llm_modal_setup/prepare_training_data.py

The parsed file is cleaned for training — fields used only at inference time are removed:

STEP_FIELDS_TO_REMOVE = {"confidence"}
PARSED_FIELDS_TO_REMOVE = {"ambiguous", "ambiguity_reason"}

Output: libero_task_mapping_parsed_training.json — this is what gets injected into the dataset and also what the inference pipeline uses for its JSON lookup.

Data Format Example

Input (English):

"pick up the black bowl on the stove and place it on the plate"

Output (compact JSON for training):

{"steps":[{"step":1,"action":"pick up","object":"bowl","color":"black","location":"stove"},{"step":2,"action":"place","object":"bowl","color":"black","location":"plate"}]}

---

11. Evaluation Results

Pretrained vs. Fine-Tuned SmolVLA (1 episode per task, JSON instructions)

Suite	Total Tasks	Pretrained (English)	Fine-Tuned (JSON)	Success Rate
LIBERO-Spatial	10	10	10	100.0%
LIBERO-Goal	10	9	9	90.0%
LIBERO-Object	10	8	8	80.0%
LIBERO-10 (long)	10	3	5	50.0%
Total	40	30 (75%)	32 (80%)	+5% improvement

The fine-tuned model matches or exceeds the pretrained baseline on all tested suites, confirming that JSON instruction fine-tuning is viable and doesn't hurt performance on simpler tasks while improving long-horizon ones.

LIBERO-90 was excluded from fine-tuning evaluation due to compute constraints (90 tasks × multiple episodes).

Evaluation Method

# Per-task eval with JSON-patched libero.py
lerobot-eval \
  --policy.path=fouad1233/smolvla_libero_json_t4 \
  --env.type=libero \
  --env.task=libero_spatial \
  --env.task_ids=[3] \
  --eval.batch_size=2 \
  --eval.n_episodes=3 \
  --policy.use_amp=true

For evaluation, the LIBERO environment's task.language field is patched at the source (libero.py) to inject the JSON string instead of English — matching the training distribution exactly.

---

12. Modal Cloud Deployment

Container Configuration

@app.cls(
    image=image,
    gpu="A100",                  # ~21 GB VRAM: Whisper(3) + MiniLM(0.5) + LLaMA(16) + SmolVLA(1.5)
    timeout=600,                 # 10 min max per pipeline run
    min_containers=0,            # Scale to zero when idle (cost saving)
    allow_concurrent_inputs=2,   # New pipeline starts even if old one is still running
)
class VoiceBotPipeline:
    ...

Container Image

The Modal image is built from debian_slim(python_version="3.12") with:

1. System libs: xvfb, ffmpeg, freeglut3-dev, libosmesa6-dev (for headless MuJoCo)
2. LeRobot cloned and installed with [libero] extras
3. Python packages: faster-whisper, sentence-transformers, transformers, accelerate, huggingface_hub
4. Local .py and .json files copied to /app in the container

Warm-Up Strategy

The local FastAPI server manages a heartbeat thread that keeps the A100 container warm between requests:

with unified_app.run():
    runner = VoiceBotPipeline()
    runner.ping.remote()          # Force cold start / model loading
    _server_status = "ready"
    while not _stop_event.is_set():
        _stop_event.wait(60)      # Ping every 60 seconds to prevent scale-down
        runner.ping.remote()

SSE Streaming Architecture

Browser  ←──── SSE ────  FastAPI (localhost:8000)  ←──── Queue ────  Modal A100
                         pipeline_server.py                           unified_modal_app.py
                         (ThreadPoolExecutor)

The worker thread calls runner.run_pipeline.remote_gen(...) which returns a Modal generator. Events are put into a Python Queue, and the async SSE generator reads from the queue and streams to the browser.

Cancel / Stop Mechanism

When the user clicks ⏹ Stop:

1. POST /pipeline/cancel/{job_id} is called
2. The stored generator reference is explicitly closed (gen.close()) — this signals Modal to terminate the remote function
3. The job queue is drained and a SENTINEL is injected
4. The SSE stream closes immediately
5. allow_concurrent_inputs=2 ensures a new pipeline can start without waiting for the old one to fully terminate on Modal's side

---

13. Web UI

File: whisper_smolvla_libero_pipeline/index.html (single file, no build step)

Features

• 4-stage pipeline visualization with live status, progress bars, and color-coded stage cards
• Microphone recording via WebRTC MediaRecorder API (requires localhost or HTTPS)
• Text chips — pre-set example commands for testing without a microphone
• Live video feed — simulation frames stream in real-time at ~12 FPS
• Match chart — top-5 cosine similarity matches visualized as a bar chart
• JSON display — the compact JSON instruction from Stage 3 is shown in the UI
• Suite filter toggles — select which LIBERO suites to search
• Stop button — cancels the current pipeline run and resets the UI
• Log console — real-time structured log from all 4 stages
• Toast notifications — success/error feedback

Running

The UI is served directly by the FastAPI server at http://localhost:8000. No build step, no npm.

⚠️ Microphone access requires localhost. Accessing via IP address will trigger browser security restrictions.

---

14. Challenges & Design Decisions

Why Modal instead of AWS?

The original proposal used Amazon-based services. Modal was adopted because it provides true serverless GPU inference — no infrastructure to manage, pay-per-second billing, and Python-native deployment. Deploying the full stack (Whisper + LLaMA + SmolVLA) as a single Modal class took ~15 minutes of setup vs. days of AWS configuration.

Why LIBERO instead of SO100 + PyBullet?

Initial experiments with the SO100 robotic arm in PyBullet with SmallVLA did not produce successful manipulation results. LIBERO provides:

• Pre-built benchmark tasks with clear success criteria
• A stable Franka Panda robot environment
• Reproducible evaluation metrics
• Integration with LeRobot's training/eval tooling

Why JSON instructions for the VLA?

SmolVLA (like most VLAs) takes a text instruction as conditioning. The standard approach is to pass the English task description. Our approach instead passes structured JSON from the LLM parser, because:

1. The LLM output is already JSON — no lossy conversion step
2. JSON is more deterministic and less ambiguous than free-form English
3. Fine-tuning on JSON improves LIBERO-10 performance

Why freeze the vision encoder during fine-tuning?

On Kaggle T4 (16 GB VRAM), freezing SigLIP (the vision encoder) reduces memory usage enough to run batch_size=4. The VLM backbone and action expert are still trained, so the model learns the JSON-to-action mapping without needing to adjust visual features.

The {} vs set() Gotcha

_cancelled_jobs: set[str] = {} creates a dict in Python (not a set). Use set() for an empty set literal. This caused a 500 error on the cancel endpoint until fixed.

---

15. References

• SmolVLA — HuggingFace, SmolVLA: Compact and Efficient Vision-Language-Action Models, 2025. HuggingFaceVLA/smolvla_libero
• LIBERO — B. Liu et al., LIBERO: Benchmarking Knowledge Transfer in Lifelong Robot Learning, NeurIPS 2023.
• LeRobot — HuggingFace, github.com/huggingface/lerobot
• LLaMA 3.1 — Meta AI, Meta-Llama-3.1-8B-Instruct, 2024.
• Whisper — OpenAI, Robust Speech Recognition via Large-Scale Weak Supervision, 2022.
• all-MiniLM-L6-v2 — Microsoft, Sentence Transformers, 2021.
• π₀ — Physical Intelligence, A Vision-Language-Action Flow Model for General Robot Control, 2024.

---

Published fine-tuned model: fouad1233/smolvla_libero_json_t4 GitHub: github.com/fouad1233/Speech_control_compatible_with_le_robot