A machine learning competition solution for predicting future price movements from Limit Order Book (LOB) data. This project explores various approaches — from gradient boosting to deep learning — to solve a challenging sequence modeling problem in high-frequency trading.
Final Submission: Two-Stage LSTM
- Public LB: 0.2754
- Private LB: 0.2944
- Rank: 85th place
| Stage | Model | Target | Description |
|---|---|---|---|
| 1 | LSTM + MLP Fusion | t0 | Sequence model with last-step MLP features |
| 2 | Simple MLP | t1 | Predicts t1 from t0 prediction |
The key insight: t0 is predictable (~0.37 correlation), while t1 has very weak signal. Using predicted t0 as a feature for t1 prediction captures the cascade relationship between targets.
| Model | t0 Score | t1 Score | Overall | Notes |
|---|---|---|---|---|
| Two-Stage LSTM | ~0.37 | ~0.07 | 0.294 | Winner - cascade approach |
| LightGBM (engineered) | 0.369 | 0.035 | 0.202 | Good t0, weak t1 |
| GRU + Attention | 0.281 | - | 0.269 | CV only, overfits |
| Ensemble (LGB+XGB+CB) | - | - | 0.202 | No diversity benefit |
┌─────────────────────────────────────────────────────────────────────────────┐
│ TWO-STAGE LSTM ARCHITECTURE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Input: Sequence of LOB states (window=100 steps × 32 features) │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ STAGE 1: T0 PREDICTION │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
│ │ │ LSTM Branch │ │ │
│ │ │ Input (100, 32) │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ LSTM(32 → 256, 1 layer) │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ Hidden State (256) │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ │ concat │ │
│ │ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────┐ │ │
│ │ │ MLP Branch │ │ │
│ │ │ Last Step (32) │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ Linear(32 → 256) + GELU │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ Linear(256 → 256) + GELU │ │ │
│ │ │ │ │ │ │
│ │ │ ▼ │ │ │
│ │ │ MLP Feature (256) │ │ │
│ │ └─────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ Fused Vector (512) │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ Dropout(0.1) + Linear(512 → 1) │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ t0_pred │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ STAGE 2: T1 PREDICTION │ │
│ │ │ │
│ │ t0_pred (1) │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ Linear(1 → 64) + GELU │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ Linear(64 → 32) + GELU │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ Linear(32 → 1) │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ t1_pred │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ Output: [t0_pred, t1_pred] clipped to [-6, 6] │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
| Approach | Result | Notes |
|---|---|---|
| Two-Stage LSTM | 0.294 Private LB | Best! Cascade: LSTM→t0, MLP→t1 |
| LSTM + MLP fusion for t0 | ~0.37 CV | Combines sequence + point features |
| Window size = 100 | Good balance | Enough context, not too slow |
| Hidden dim = 256 for t0 | Optimal | Larger didn't help |
| Simple MLP for t1 | ~0.07 | t0 prediction is the key feature |
| Approach | Result | Why It Failed |
|---|---|---|
| Gradient Boosting (engineered features) | 0.202 | Good t0, but t1 is nearly random |
| GRU with attention | 0.269 CV | Overfits, worse on test |
| Ensemble (LGB+XGB+CB) | 0.202 | Models too correlated |
| Transformer encoder | <0.20 | Too many parameters |
| AutoML (FEDOT, H2O) | <0.15 | Not suited for this problem |
| Prophet | ~0.00 | Designed for trends, not LOB |
| Predicting t1 directly | ~0.03 | Signal too weak |
| Independent t0/t1 models | Lower | Missing cascade relationship |
-
Cascade relationship matters: t1 benefits from knowing t0 predictions, even if t0 is imperfect
-
t1 signal is extremely weak: Direct prediction fails; need indirect approach
-
Simple is better: 1-layer LSTM + simple MLP outperforms complex architectures
-
Feature engineering ≠ learned features: For this problem, LSTM learned better temporal features than handcrafted ones
-
Small dataset challenge: With only 517 sequences, overfitting is inevitable — success depends on generalization strategies
lob-predictorium/
├── src/
│ ├── data/
│ │ ├── loader.py # Data loading utilities
│ │ ├── features.py # Batch & online feature engineering
│ │ └── dataset.py # PyTorch dataset for sequences
│ ├── models/
│ │ ├── boosting.py # LightGBM/XGBoost/CatBoost wrapper
│ │ ├── ensemble.py # Ensemble blending with Optuna
│ │ └── deep/
│ │ ├── gru_model.py # GRU with attention
│ │ ├── gru_dual_branch.py # Dual-branch GRU
│ │ ├── lstm_dual_branch.py # Dual-branch LSTM (winning)
│ │ ├── two_stage.py # Two-stage deep model
│ │ ├── transformer.py # Transformer encoder
│ │ ├── losses.py # Custom loss functions
│ │ └── train.py # Training utilities
│ ├── evaluation/
│ │ ├── scorer.py # Weighted Pearson implementation
│ │ └── cv.py # Cross-validation utilities
│ ├── submission/ # Boosting-based submission (backup)
│ │ └── solution.py
│ └── submission_two_stage/ # ⭐ Best submission (Two-Stage LSTM)
│ ├── solution.py # Self-contained prediction model
│ ├── t0_model.pt # Trained LSTM for t0
│ └── t1_model.pt # Trained MLP for t1
├── scripts/
│ ├── train_t0_lstm.py # Stage 1: Train t0 LSTM
│ ├── train_t1_from_t0.py # Stage 2: Train t1 MLP
│ ├── train_boosting.py # Train boosting models
│ ├── train_deep.py # Train GRU/LSTM/Transformer
│ ├── build_ensemble.py # Build model ensembles
│ ├── feature_selection.py # Feature selection
│ ├── evaluate.py # Evaluate trained models
│ └── submit.py # Package submission zip
├── configs/
│ └── *.yaml # Model configurations
├── docs/
│ ├── ARCHITECTURE.md # Detailed architecture docs
│ └── faq.md # Competition FAQ
└── artifacts/ # Model checkpoints (gitignored)
The competition submission environment has strict resource limits:
| Resource | Limit |
|---|---|
| CPU | 1 core |
| RAM | 16 GB |
| GPU | None (CPU inference only) |
| Time limit | 60 minutes for entire test set |
| Test set size | ~1,500 sequences |
Implications:
- Models must be optimized for CPU inference
- No GPU-dependent architectures (bidirectional RNNs, large transformers)
- ONNX runtime recommended for faster inference
- Simple architectures often outperform complex ones due to latency constraints
# Clone the repository
git clone https://github.com/yourusername/lob-predictorium.git
cd lob-predictorium
# Install dependencies (requires uv)
uv sync
# Or with pip
pip install -e .# Train Two-Stage LSTM (best model)
# Stage 1: Train t0 LSTM
uv run python scripts/train_t0_lstm.py
# Stage 2: Train t1 MLP using t0 predictions
uv run python scripts/train_t1_from_t0.py --t0-checkpoint artifacts/deep/t0_lstm/fold_2
# Alternative: Train boosting model
uv run python scripts/train_boosting.py --model lightgbmuv run python scripts/evaluate.py --model-type deep --model-path artifacts/deep/t0_lstm# Package Two-Stage LSTM submission
uv run python scripts/submit.py --solution-dir src/submission_two_stage --output two-stage-lstm.zipStage 1: T0 LSTM Forecaster
T0OnlyLSTMForecaster(
input_dim=32,
hidden_dim=256,
dropout=0.1,
)
# LSTM: 32 → 256 (1 layer)
# MLP on last step: 32 → 256 → 256
# Fusion: concat(lstm_hidden, mlp) → 512 → 1Stage 2: T1 MLP Predictor
T1FromT0Predictor(
hidden_dim=64,
)
# MLP: 1 → 64 → 32 → 1Training Parameters
- Window size: 100 steps
- Batch size: 64
- Learning rate: 3e-5 (t0), 1e-3 (t1)
- Early stopping patience: 5 (t0), 10 (t1)
- AMP enabled
The submission must predict step-by-step with a sliding window:
class PredictionModel:
def __init__(self):
self.buffer = np.zeros((100, 32)) # Circular buffer
self.t0_model = T0OnlyLSTMForecaster(...)
self.t1_model = T1FromT0Predictor(...)
def predict(self, data_point: DataPoint) -> np.ndarray | None:
# Update buffer
self.buffer[pos] = data_point.state
if not data_point.need_prediction:
return None
# Stage 1: Predict t0
window = self._get_window() # (100, 32)
t0_pred = self.t0_model(window)
# Stage 2: Predict t1 from t0
t1_pred = self.t1_model(t0_pred)
return np.array([t0_pred, t1_pred])- t0 has strong sequential signal: LSTM captures temporal patterns effectively
- t1 benefits from t0: Even imperfect t0 predictions help t1 modeling
- Specialization: Each model focuses on one target with appropriate architecture
- Avoid overfitting: Simple MLP for t1 prevents overfitting to weak signal
| Technique | Impact | Notes |
|---|---|---|
| SWA | +0.002-0.003 | Average weights of last 3-5 epochs |
| Chrono initialization | Significant | Better long-term memory in LSTM/GRU |
| Two-stage boosting | Moderate | Second model predicts residual error |
| Data augmentation | 5x more data | Variance normalization + stretch/compress |
| Masked autoencoder | Good | Pretrain on full data, fine-tune on train |
- FiLM ensemble: Feature-wise Linear Modulation for model fusion
- Adaptive weighting: Balance model weights by per-sequence MSE
- Diversity trick: Combine overfitted + generalized models
- Quantization: ONNX + Int8 to fit 4 models under 20MB
- Highway heads instead of linear layers
- LSTM + Self-Attention on last hidden state (mitigates decay)
- Self-Attention + GeGLU MLP-Mixer encoder
- Bidirectional LSTM over last 60 steps (weak metric model)
- Classic transformer often outperformed specialized (TFT, Informer, Mamba)
- Ranger (RAdam + lookahead) — worked well for transformers
- Muon, SOAP — next-gen whitening matrix optimizers
The main challenge is generalization with only 517 sequences. SWA and ensemble techniques provided the biggest boosts. Classic architectures with proper training strategies often beat complex specialized models.
MIT License — feel free to use and modify for your own experiments.