This example runs the Hendrycks Sanity Check experiment proposed in Defeating the Training-Inference Mismatch. The sanity check tests whether an RL algorithm can reliably improve a model on problems it can already partially solve. The dataset is filtered from MATH to only include problems where the base model (DeepSeek-R1-Distill-Qwen-1.5B) solves 20-80% of the time across 40 rollouts. A reliable algorithm should push training accuracy on this "perfectible" subset above 95%.
Because our trainer is asynchronous, we perform only one gradient step per batch (the inference engine generates the next batch while the trainer processes the current one).
This example runs on 8 GPUs (4 for inference, 4 for training).
Schedule training locally on a node with 8 GPUs
uv run rl @ examples/basic/hendrycks-sanity/rl.toml \
--wandb.project your-project \
--wandb.name your-run