EDA, feature engineering, and model evaluation used to prepare and validate the ML pipeline for a production taxi fare and duration prediction service.
This repository contains exploratory data analysis (EDA), feature engineering, and model evaluation work that fed into a microservices-based NYC taxi fare and trip-duration prediction application. The resulting feature set and modeling approach were used by the model service in the broader project (FastAPI backend, React/Streamlit frontends, OSRM routing, Redis, PostgreSQL).
- Problem Statement
- Dataset Description
- Modeling Approach
- Evaluation Metrics
- Results
- How to Run
- Architecture Diagram
- Project Structure
- Technologies Used
The production system must predict two targets for NYC yellow taxi trips:
- Trip duration (seconds) — to support ETA and scheduling.
- Fare amount (dollars) — to support pricing and demand estimation.
Predictions are made at request time using trip and context features (pickup/dropoff, time, passenger count, distance, holiday, weather, etc.). This repository addresses the upstream work required to:
- Explore and clean NYC TLC trip data and external enhancement sources (weather, events, holidays).
- Define and implement a repeatable feature-engineering pipeline (time, weather, holiday, location heat, events, sports) that can be mirrored in the model service.
- Compare baseline and tuned models (e.g., LightGBM) for duration and fare using a chained setup: duration is predicted first, then duration predictions are used as an input feature for fare prediction.
- Evaluate and document which feature groups improve validation performance (RMSE, R²) and support selection of the final feature set and hyperparameters for the production model service.
The deliverables are cleaned datasets, feature definitions, training/evaluation code, and validation metrics that justify the choices used in the deployed service.
Trip data is sourced from the NYC TLC Trip Record Data. Parquet files are read directly from the official CDN:
- Single-month (e.g., May 2022):
https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2022-05.parquet - Six-month (Jan–Jun 2022):
Same base URL withyellow_tripdata_2022-01.parquetthroughyellow_tripdata_2022-06.parquet.
Notebooks typically use a 10% sample (sample(frac=0.1, random_state=42)) for faster iteration. Key columns include pickup/dropoff datetime and location IDs, trip distance, passenger count, fare amount, duration, and related fields used for cleaning and feature construction.
| Source | Purpose | Notes |
|---|---|---|
| Weather | weather_code for merge with trips by date/hour |
NOAA CDO API (GHCND, Central Park) or CSV fallback; see weather_integration.py. |
| US public holidays | is_holiday |
holidays library (US) used in feature_generation.py. |
| Events (Ticketmaster / NYC Parks) | Event proximity and count features | Optional; Ticketmaster Discovery API or NYC Parks CSV; see event_data_integration.py and preprocess_enhancement_data.py. |
| Sports schedules | Game-day and venue-proximity features | Optional; CSV with date, venue, optional start time; see event_data_integration.py. |
| Zone centroids | Map PULocationID to lat/lon for distance-based event/sports features |
Optional CSV; see event_data_integration.py. |
Weather is required for the main pipelines; event and sports data are optional and used when paths are configured in the notebooks.
- Load and clean trip data — Filter to 2022, drop nulls and implausible fare/duration, optional outlier clipping (e.g., 99th percentile).
- Enhance with external data — Join weather (date/hour); optionally add holiday, events, and sports features.
- Feature engineering — Time (e.g.,
pickup_tod,pickup_dow,pickup_month), weather, holiday, location heat (pickup/dropoff zone frequency), and optional event/sports features. Categoricals are one-hot encoded (e.g.,weather_code,rate_code_id,payment_type); drop-first is used where appropriate. - Train/validation split — Time-based or random 70/30 (or similar) as in the notebooks.
- Duration model — Trained on the chosen feature set; validation metrics (RMSE, MAE, R²) are computed.
- Fare model — Uses the same features plus predicted duration (and optionally OOF duration predictions when applicable). Validation metrics are computed for fare.
- Hyperparameter tuning — GridSearchCV (or similar) over LightGBM parameters; baseline vs. tuned comparison is done in the fully integrated notebooks.
The primary model is LightGBM (Gradient Boosting). The codebase also supports Linear Regression, Random Forest, Decision Tree, XGBoost, and Gradient Boosting for experimentation; the production-oriented notebooks use LightGBM for both duration and fare.
Fare is modeled as a function of trip and context features plus predicted duration. This reflects the real-world dependency of fare on trip length and time. The notebooks implement this either by adding a column of duration predictions on the validation set or by using an out-of-fold duration prediction pipeline where applicable.
Model performance is evaluated with standard regression metrics:
| Metric | Description |
|---|---|
| MSE | Mean squared error. |
| RMSE | Root mean squared error (same units as target: seconds for duration, dollars for fare). |
| MAE | Mean absolute error. |
| R² | Coefficient of determination (0–1 scale; higher is better). |
These are computed on the validation (or test) set after training. The notebooks and regression_models.py report MSE, RMSE, MAE, and R²; model_evaluation.py provides plotting helpers to compare baseline vs. tuned models (e.g., bar charts for RMSE and R²).
Validation results from the fully integrated pipeline (e.g., 6-month data, base + time + weather + holiday + heat + events) are representative of what is documented in the notebooks:
-
Duration (seconds)
- Baseline LightGBM: RMSE ~257.5, R² ~0.82.
- Tuned LightGBM: RMSE ~236.8, R² ~0.85.
-
Fare (dollars)
- Baseline LightGBM: RMSE ~1.97, R² ~0.96.
- Tuned LightGBM: RMSE ~1.86, R² ~0.97.
Adding the full feature set (time, weather, holiday, heat, and optional event/sports features) and tuning improves both duration and fare validation metrics compared to a baseline feature set. Feature importance plots (via model_evaluation.plot_feature_importance) are used to inspect which drivers matter most for duration and fare. These results support the feature set and model configuration used in the production model service.
- Python 3.9, 3.10, 3.11, or 3.12
- Git (for cloning the repository)
Recommended: use a virtual environment and install dependencies from the main project (e.g., pandas, numpy, scikit-learn, lightgbm, xgboost, matplotlib, python-dotenv, requests, holidays). If the parent repo has a requirements.txt, install from that; otherwise install these packages as needed. For weather API and optional event APIs, set the appropriate environment variables (e.g., NOAA_CDO_TOKEN, TICKETMASTER_API_KEY) or rely on CSV fallbacks where documented.
git clone <repository-url>
cd "<path-to-sprint5-FE>"
python -m venv venv
# Windows (PowerShell):
.\venv\Scripts\Activate.ps1
# Linux/macOS:
# source venv/bin/activate
pip install --upgrade pip
pip install pandas numpy scikit-learn lightgbm xgboost matplotlib python-dotenv requests holidays-
Start Jupyter from the repository root (so that
importfromevent_data_integration,weather_integration,feature_generation, etc. works):jupyter notebook
-
Open notebooks in order:
- 0_EDA_and_baseline.ipynb — Initial EDA and baseline model.
- 1_FE_DA_looping_selection.ipynb — Iterative feature selection (single-month).
- 2_FE_DA_prediction_feeding_pipeline.ipynb — Chained duration-to-fare pipeline.
- 3_FE_DA_fully_integrated.ipynb — Full pipeline with baseline vs. tuned comparison (single-month).
- 4_EDA_and_baseline_6months.ipynb, 5_FE_DA_looping_selection_6months.ipynb, 6_FE_DA_fully_integrated_6months.ipynb — Same flow over six months of data.
-
Ensure the kernel can see the local modules (
event_data_integration,weather_integration,preprocess_enhancement_data,feature_generation,FE_DA_helper_functions,regression_models,model_evaluation). If needed, add the project root tosys.path(e.g.,sys.path.insert(0, ".")) in the first cell. -
Optional: set
MLFLOW_DIRand run with MLflow tracking (as in the notebooks) to log experiments.
The pipeline logic lives in the .py modules. The notebooks orchestrate loading data, calling preprocessing and feature generation, splitting, training (e.g., via FE_DA_helper_functions.train_and_evaluate_model_loop and regression_models), and evaluation. To run a similar flow from a script, import from these modules and call the same functions in sequence; refer to the fully integrated notebooks for the exact order of operations.
+------------------------------------------------------------------+
| DATA SOURCES |
+------------------------------------------------------------------+
| |
| +----------------------+ +----------------------------------+ |
| | NYC TLC Parquet | | Enhancement data | |
| | (Yellow Taxi trips) | | Weather (NOAA/CSV), Holidays, | |
| | 2022-01 .. 2022-06 | | Events (Ticketmaster/Parks), | |
| | | | Sports schedules, Zone centroids | |
| +----------------------+ +----------------------------------+ |
| | | |
+------------+------------------------------+------------------------+
| |
v v
+------------------------------------------------------------------+
| PREPROCESSING & FEATURE ENGINEERING |
| preprocess_enhancement_data.py | feature_generation.py |
| event_data_integration.py | weather_integration.py |
+------------------------------------------------------------------+
| - Clean trip data (nulls, sanity checks, optional outlier clip) |
| - Merge weather by (date, hour); add holiday, time, heat |
| - Optional: event/sports features, zone centroids for lat/lon |
| - One-hot encode categoricals; drop unneeded columns |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| TRAIN / VALIDATION SPLIT |
| FE_DA_helper_functions.py |
+------------------------------------------------------------------+
| - train_test_split_loop, preprocess_for_modeling_loop |
| - Encode and prepare feature matrix X and targets (duration, |
| fare) for multiple dataset variants (base, base+time, etc.) |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| MODEL TRAINING & TUNING |
| regression_models.py |
+------------------------------------------------------------------+
| - Duration: train_and_evaluate_model_mlflow / train loop |
| - Fare: same, with predicted duration (or OOF) as feature |
| - tune_lightgbm_cv for hyperparameter search |
| - evaluate_regression_metrics (MSE, RMSE, MAE, R2) |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| EVALUATION & VISUALIZATION |
| model_evaluation.py |
+------------------------------------------------------------------+
| - get_lightgbm_from_fitted_list, plot_feature_importance |
| - plot_baseline_vs_tuned (RMSE, R2 comparison) |
+------------------------------------------------------------------+
|
v
+------------------------------------------------------------------+
| OUTPUT |
+------------------------------------------------------------------+
| - Validation metrics (RMSE, R2, etc.) for baseline vs. tuned |
| - Feature importance plots |
| - MLflow runs (if enabled) |
| - Informed feature set and hyperparameters for production model |
+------------------------------------------------------------------+
Flow summary: raw trip and enhancement data are preprocessed and merged; features are engineered and encoded; data are split and used to train duration and fare models (with duration feeding into fare); models are tuned and evaluated; results support the production model service configuration.
sprint5-FE/
|-- 0_EDA_and_baseline.ipynb # Initial EDA and baseline
|-- 1_FE_DA_looping_selection.ipynb # Feature selection (single-month)
|-- 2_FE_DA_prediction_feeding_pipeline.ipynb # Duration -> fare pipeline
|-- 3_FE_DA_fully_integrated.ipynb # Full pipeline, baseline vs tuned (single-month)
|-- 4_EDA_and_baseline_6months.ipynb # EDA and baseline (6 months)
|-- 5_FE_DA_looping_selection_6months.ipynb # Feature selection (6 months)
|-- 6_FE_DA_fully_integrated_6months.ipynb # Full pipeline (6 months)
|-- FE_DA_original.ipynb # Original feature-engineering draft
|-- event_data_integration.py # Events, sports, zone centroids, distance features
|-- feature_generation.py # Time, weather, holiday, heat, duration-copy features
|-- FE_DA_helper_functions.py # Batch preprocessing, encode, split, train/eval loops
|-- model_evaluation.py # LightGBM extraction, importance plot, baseline vs tuned plot
|-- preprocess_enhancement_data.py # Standardize and clean weather, events, sports
|-- regression_models.py # Train/eval, metrics, tuning, OOF/duration-for-fare
|-- weather_integration.py # NOAA weather fetch and CSV fallback
|-- README.md # This file
- Python 3.9+: Core language.
- Pandas / NumPy: Data loading, cleaning, and feature construction.
- Scikit-learn: Train/test split, metrics (MSE, RMSE, MAE, R²), GridSearchCV, and supporting models.
- LightGBM / XGBoost: Primary and alternative gradient boosting regressors.
- Matplotlib: Diagnostic and comparison plots.
- Jupyter: Interactive notebooks for EDA and pipeline runs.
- python-dotenv / requests: Environment variables and API calls for weather and optional event APIs.
- holidays: US public holiday calendar for
is_holidayfeature. - MLflow (optional): Experiment tracking when enabled in the notebooks.
This work is part of a larger NYC taxi fare and duration prediction project.
Marissa Singh