Predict tomorrow's rain and forecast hourly temperature for Azamgarh, Uttar Pradesh using ERA5 reanalysis data, XGBoost, and Bidirectional LSTM — served via a REST API.

---

📋 Table of Contents

• Overview
• LiveDemo
• Screenshots
• Architecture
• Features
• Tech Stack
• Project Structure
• Installation
• API Documentation
• Pipeline Overview
• Deployment
• Dataset
• Models & Performance
• Roadmap
• Contributing
• License
• Author
• Acknowledgements

---

📖 Overview

This project is an end-to-end Weather Forecasting System that combines Machine Learning, Deep Learning, and real-time weather data to provide accurate rainfall predictions and short-term temperature forecasts for Azamgarh, Uttar Pradesh, India (26.04°N, 83.11°E).

Historical weather observations are collected from the ERA5 reanalysis dataset provided by ECMWF, while live weather conditions are retrieved from the Open-Meteo API. Together, these data sources enable the system to generate real-time forecasts without relying on a local weather database.

The application exposes a FastAPI REST API and an interactive Streamlit dashboard, making predictions easily accessible for both developers and end users.

🎯 Prediction Tasks

Forecast	Model	Learning Task	Output
🌧️ Rain Prediction	XGBoost Classifier	Binary Classification	Rain probability and tomorrow's rain prediction
🌡️ Temperature Forecast	BiLSTM Encoder–Decoder	Multi-step Time Series Forecasting	Hourly temperature forecast for the next N hours

🌍 Data Sources

• ERA5 Reanalysis Dataset (ECMWF) — historical weather data used for model training.
• Open-Meteo API — provides real-time weather observations and forecast data for live inference.

Why ERA5?
ERA5 is one of the world's most trusted climate reanalysis datasets. It provides high-quality, spatially consistent, and gap-free weather observations, including atmospheric variables such as temperature, pressure, humidity, wind speed, and multi-layer cloud cover, making it well suited for machine learning-based weather forecasting.

✨ Key Features

• 🌦️ Real-time weather forecasting using the Open-Meteo API
• 🤖 XGBoost model for rainfall prediction with probability estimation
• 🧠 BiLSTM Encoder–Decoder model for multi-step temperature forecasting
• 📈 Advanced feature engineering with lag features, rolling statistics, and cyclical time encoding
• 🔄 Automated ERA5 data collection pipeline using the Copernicus CDS API
• ⚡ High-performance REST API built with FastAPI
• 🖥️ Interactive Streamlit dashboard for visualization
• 📊 Built-in model evaluation and metadata endpoints
• 🔁 Easily configurable for any geographic location through environment variables

---

🚀 Live Demo

Service	URL
🎨 Live App	https://azamgarh-weather-forecasting.onrender.com
📖 API Docs	/docs · /redoc
⚡ Health Check	https://azamgarh-weather-forecasting.onrender.com/health

Note: The backend is hosted on Render's free tier and may take 30–60 seconds to wake up on first request.

---

📸 Screenshots

🏠 Home — Interactive Weather Dashboard

📘 API Documentation — FastAPI Interactive Docs

🌦️ Weather Forecast — Temperature, Rain Prediction & 7-Day Forecast

📊 Model Performance — Evaluation Metrics & Parameters

📥 Data Collection Pipeline

🗺️ Open-Meteo Feature Mapping

---

🏗️ Architecture

Full System Architecture

---

✨ Features

	Feature	Description
📥	Automated ERA5 Pipeline	Monthly downloads via the Copernicus CDS API
🔧	Feature Engineering	Lag features, rolling averages, cyclical (sin/cos) encodings
📊	Comprehensive EDA	17 publication-quality figures across 11 analysis sections
🌲	XGBoost Classifier	Time-series CV, class imbalance handling, threshold tuning
🧠	BiLSTM Encoder-Decoder	Configurable N-step temperature forecasting
🌐	REST API	Clean, documented endpoints with JSON responses
📡	Open-Meteo Integration	Live inference with no database or scheduled job required
🧪	Calibrated Predictions	Probability outputs with confidence levels
📦	Reproducible	One .env file controls the entire system

---

🛠️ Tech Stack

Backend

• FastAPI — high-performance REST API framework
• Uvicorn — ASGI server
• Pydantic — request and response validation
• python-dotenv — environment variable management
• Joblib — model serialization and loading

Machine Learning

• XGBoost — temperature forecasting model
• TensorFlow / Keras — rainfall prediction model
• Scikit-learn — preprocessing, feature engineering, and model evaluation

Data Processing

• Pandas — data manipulation and analysis
• NumPy — numerical computing

Data Visualization

• Matplotlib — exploratory data visualization
• Seaborn — statistical plotting

Data Sources

• ERA5 via cdsapi — historical weather data for model training
• Open-Meteo API — real-time weather data and forecasts

Frontend

• Requests — communication with the FastAPI backend

---

📂 Project Structure

weather-forecasting-system/
│
├── app/
│   ├── routes/
│   │   └── weather_routes.py       # API route definitions
│   ├── services/
│   │   ├── forecast_service.py     # Forecast business logic
│   │   ├── ml_services.py          # ML inference service
│   │   ├── model_loader.py         # Model loading & caching
│   │   └── weather_service.py      # Weather data service
│   └── utils/
│       └── weather_utils.py        # Utility/helper functions
│
├── config.py                       # App configuration (env, paths, settings)
├── main.py                         # Application entry point
│
├── data/
│   ├── archives/                   # Monthly zipped raw data archives
│   │   ├── 2021/ … 2025/           # azamgarh_weather_<MONTH>_<YEAR>.zip
│   │
│   ├── processed/                  # Cleaned and feature-engineered CSVs
│   │   ├── azamgarh_weather_clean.csv
│   │   ├── azamgarh_weather_final.csv
│   │   └── azamgarh_weather_raw.csv
│   │
│   └── raw/                        # Raw ERA5 data streams (by year/month)
│       └── 2021/ … 2025/
│
├── models/
│   ├── lstm/                       # LSTM temperature model
│   │   ├── best_model.keras
│   │   ├── lstm_temp_model.keras
│   │   └── model_meta.json
│   ├── scalers/
│   │   └── lstm_temp_scaler.pkl    # Fitted scalers for normalization
│   └── xgboost/                    # XGBoost rainfall model
│       ├── model_meta.json
│       └── rain_model.json
│
├── notebooks/                          # Jupyter notebooks for EDA & training
│   ├── eda_figures/                    # Saved EDA plots
│   ├── 01_era5_eda.ipynb               # ERA5 exploratory data analysis
│   ├── 03_xgboost_train.ipynb          # XGBoost model training
│   ├── 04_lstm_train.ipynb             # LSTM model training
│   ├── download_weather.py             # Script to download weather data
│   └── weather_dataset_pipeline.ipynb
│
├── pipeline/                             # End-to-end ML pipeline scripts
│   ├── 01_download_weather.py            # Step 1: Download ERA5 data
│   ├── 02_weather_dataset_pipeline.ipynb # Step 2: Extract ZIPs and merge
│   ├── 03_preprocess.py                  # Step 3: Clean & preprocess data
│   ├── 04_feature_engineer.py            # Step 4: Feature engineering
│   ├── 05_train_xgboost.py               # Step 5: Train XGBoost model
│   └── 06_train_lstm.py                  # Step 6: Train LSTM model
│
├── static/
│   ├── script.js                   # Frontend JavaScript
│   └── style.css                   # Frontend styles
│
├── templates/
│   └── index.html                  # Main HTML template
│
├── images/                         # README assets (screenshots, banner, diagrams)
├── .env                            # Environment variables (not committed)
├── .env.example                    # Template so others can set up the project
├── .gitignore
├── requirements.txt                 # Python dependencies
└── README.md

---

⚙️ Installation

1. Clone the repository

git clone https://github.com/sharif-abusad/weather-forecasting-system.git
cd weather-forecasting-system

2. Create and activate a virtual environment

# Windows
python -m venv .venv
.venv\Scripts\activate

# Linux / Mac
python -m venv .venv
source .venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

4. Set up environment variables

cp .env.example .env
# edit .env with your CDS API key and other config

5. Set up CDS API credentials

Create ~/.cdsapirc in your home directory:

url: https://cds.climate.copernicus.eu/api/v2
key: YOUR-UID:YOUR-API-KEY

Get your UID and API key from your CDS profile page.

6. Run the full training pipeline (skip if models are already trained)

python pipeline/01_download_weather.py
python weather_dataset_pipeline.py
python pipeline/03_preprocess.py
python pipeline/04_feature_engineer.py
python pipeline/05_train_xgboost.py
python pipeline/06_train_lstm.py

7. Start the API

python main.py

Open http://localhost:5000 in your browser.

---

📡 API Documentation

The Weather Forecasting API provides real-time weather insights, rainfall prediction, system health monitoring, and model metadata.

---

GET /weather/{city}

Returns the current weather conditions, a 5-hour temperature forecast, a 5-day weather forecast, and tomorrow's rainfall prediction for the specified city.

Example Request

GET /weather/Azamgarh

Example Response

{
  "current": {
    "temperature": 33.4,
    "feels_like": 38.4,
    "humidity": 64.91,
    "surface_pressure": 985.7,
    "wind_speed": 15.4,
    "city": "Azamgarh, Uttar Pradesh, India"
  },
  "hourly": [
    {
      "time": "17:00",
      "temp": 31
    },
    {
      "time": "18:00",
      "temp": 30
    }
  ],
  "forecast": [
    {
      "date": "Sun, Jul 05",
      "temp_min": 29,
      "temp_max": 36,
      "description": "Unknown"
    }
  ],
  "rain_prediction": {
    "rain_tomorrow": false,
    "probability": 0.5187,
    "confidence": "low (uncertain)"
  }
}

---

GET /model-meta

Returns detailed metadata, training configuration, evaluation metrics, and hyperparameters for both machine learning models.

Included Information

🌡️ BiLSTM Temperature Forecasting Model

• Model architecture
• Input features
• Lookback window
• Forecast horizon
• Training configuration
• Dataset statistics
• Overall evaluation metrics (MAE, RMSE, R²)
• Per-hour forecasting performance (+1h to +5h)

🌧️ XGBoost Rainfall Prediction Model

• Feature list
• Hyperparameters
• Cross-validation results
• ROC-AUC and Average Precision
• Accuracy & F1 Score
• Optimal classification threshold
• Feature importance rankings

Example Response

{
  "lstm": {
    "model": "BiLSTM Encoder-Decoder",
    "metrics": {
      "mae_celsius": 1.0912,
      "rmse_celsius": 1.3169,
      "r2_score": 0.9601
    }
  },
  "xgboost": {
    "model": "XGBoostClassifier",
    "roc_auc": 0.9641,
    "avg_precision": 0.9283,
    "metrics_at_best_threshold": {
      "accuracy": 0.9061,
      "f1_score": 0.8532
    }
  }
}

---

GET /health

Checks whether the API is running correctly and confirms that the trained models have been loaded successfully.

Example Response

{
  "status": "healthy",
  "models_loaded": true
}

---

GET /

Serves the Streamlit Weather Forecasting Dashboard.

---

API Overview

Method	Endpoint	Description
GET	/weather/{city}	Returns current weather, rainfall prediction, hourly temperature forecast, and multi-day weather forecast
GET	/model-meta	Returns model architecture, hyperparameters, training details, and evaluation metrics
GET	/health	Health check endpoint for API and model status
GET	/	Launches the Streamlit Weather Forecasting dashboard

---

🔄 Pipeline Overview

                 ┌──────────────┐
                 │  ERA5 CDS    │
  Trigger ─────▶│   API Call   │  01_download_weather.py
                 └──────┬───────┘
                        ▼
                 ┌──────────────┐
                 │ Build Dataset│  02_weather_dataset_pipeline.ipynb
                 │ (extract +   │  (unzip, merge months → raw CSV)
                 │  merge ZIPs) │
                 └──────┬───────┘
                        ▼
                 ┌──────────────┐
                 │  Preprocess  │  03_preprocess.py
                 │ (clean, IST  │
                 │  conversion) │
                 └──────┬───────┘
                        ▼
                 ┌──────────────┐
                 │   Feature    │  04_feature_engineer.py
                 │  Engineering │  (lags, rolling, cyclical encoding)
                 └──────┬───────┘
                        │
          ┌─────────────┴─────────────┐
          ▼                           ▼
   ┌─────────────┐            ┌─────────────┐
   │ 05_train_   │            │ 06_train_   │
   │ xgboost.py  │            │ lstm.py     │
   │ (rain model)│            │ (temp model)│
   └──────┬──────┘            └──────┬──────┘
          └─────────────┬─────────────┘
                        ▼
                 ┌──────────────┐
                 │  Fast REST   │  live requests →
                 │     API      │  Open-Meteo → predictor.py → JSON
                 └──────────────┘

---

☁️ Deployment

Docker

Add a Dockerfile similar to the one below if you haven't already:

FROM python:3.10-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 5000
CMD ["python", "main.py"]

docker build -t weather-forecasting-system .
docker run -p 5000:5000 --env-file .env weather-forecasting-system

Render

1. Create a Web Service from this repository.
2. Build command: pip install -r requirements.txt
3. Start command: uvicorn app.main:app --reload
4. Add CDS_UID, CDS_API_KEY, and the other .env variables under Render's Environment settings.

Notes

• The training pipeline (ERA5 downloads, XGBoost/LSTM training) is meant to run offline, ahead of deployment — only the trained model artifacts under models/ need to ship with the deployed API.
• Add automated retraining (see Roadmap) before relying on this in a long-running production setting, since ERA5-derived models can drift as seasons change.

---

📊 Dataset

Weather data is sourced from ERA5 (ECMWF Reanalysis v5) via the Copernicus Climate Data Store (CDS) for Azamgarh, Uttar Pradesh, India.

Property	Value
Source	ERA5 Reanalysis (ECMWF) via Copernicus CDS
Location	Azamgarh, UP, India (26.04°N, 83.11°E)
Period	January 2021 – December 2025
Frequency	Hourly
Total rows	~43,800
Features	18 columns (12 numeric + 5 temporal + 1 target)
Missing values	None
Download format	NetCDF → CSV

Feature Descriptions

Column	Unit	Description
valid_time	DateTime	Hourly timestamp (IST, UTC+5:30)
temperature	°C	Air temperature at 2m above surface
surface_pressure	hPa	Atmospheric pressure at surface
total_cloud_cover	0–1	Total cloud fraction
low_cloud_cover	0–1	Cloud fraction below 2 km
medium_cloud_cover	0–1	Cloud fraction 2–6 km
high_cloud_cover	0–1	Cloud fraction above 6 km
precipitation	mm	Hourly accumulated precipitation
humidity	%	Relative humidity at 2m
wind_speed	m/s	Wind speed at 10m
hour	0–23	Hour of day
month	1–12	Month of year
temp_rolling_6	°C	6-hour rolling mean temperature
temp_lag_1	°C	Temperature 1 hour ago
temp_lag_24	°C	Temperature 24 hours ago
rain_tomorrow	0/1	Target: rain in next 24h > 0.1mm

---

🤖 Models & Performance

XGBoost Rain Classifier

Parameter	Value
n_estimators	300
max_depth	5
learning_rate	0.05
subsample	0.8
colsample_bytree	0.8
eval_metric	logloss
early_stopping_rounds	20
Class imbalance	sample_weight='balanced'
Validation	5-fold TimeSeriesSplit

Metric	Value
ROC-AUC	0.9641
F1-Score	0.8532
Accuracy	0.9061
Avg Precision	0.9283
Best Threshold	0.7
CV ROC-AUC (mean ± std)	0.9126 0.0558

Bidirectional LSTM

Parameter	Value
Architecture	Encoder-Decoder
Encoder	BiLSTM (64) → Dropout → LSTM (32) → BatchNorm
Bridge	RepeatVector (N)
Decoder	LSTM (32) → Dropout → BatchNorm → TimeDistributed Dense
Loss	Huber
Optimizer	Adam (lr=1e-3)
Lookback	24 hours (configurable via LOOKBACK)
Forecast horizon	5 hours (configurable via FORECAST_N)
Trainable params	~66,593

Horizon	MAE (°C)	RMSE (°C)	R²
+1h	0.9509	1.1245	0.9708
+2h	1.0527	1.2477	0.9641
+3h	1.0931	1.3121	0.9604
+4h	1.17	1.4183	0.9538
+5h	1.1891	1.4549	0.9514
Overall	1.0912	1.3169	0.9601

Prerequisites

Requirement	Version	Notes
Python	3.10+
pip	23.0+
CDS Account	Free	Register here
RAM	8 GB+	For LSTM training
GPU	Optional	CUDA 11.8+ for faster LSTM training

---

🗺 Roadmap

• ERA5 data pipeline (monthly downloads)
• Feature engineering (lags, rolling, target)
• EDA notebook (17 figures)
• XGBoost classifier with threshold tuning
• BiLSTM encoder-decoder temperature forecasting
• Flask REST API
• Open-Meteo live integration
• Sin/cos cyclical encoding for hour and month
• Add humidity to the LSTM feature set
• Extend LSTM forecast horizon to 24 hours
• Deploy to cloud (Render / Railway / AWS)
• Add automated daily retraining pipeline
• Add precipitation amount regression model
• Compare against a Transformer-based model (PatchTST)
• Multi-location support across eastern UP

---

🤝 Contributing

Contributions are welcome.

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature-name
3. Commit your changes: git commit -m 'Add: your feature description'
4. Push to your branch: git push origin feature/your-feature-name
5. Open a Pull Request

---

📄 License

This project is licensed under the MIT License.

---

👤 Author

Sharif Abusad

If you found this project useful, consider giving it a ⭐ on GitHub — it helps a lot!

---

🙏 Acknowledgements

• ECMWF / Copernicus — for making ERA5 reanalysis data freely available via the Climate Data Store
• Open-Meteo — for the free weather API that shares ERA5 data lineage
• XGBoost team — Chen & Guestrin (2016) for the XGBoost algorithm
• TensorFlow / Keras team — for the deep learning framework
• Scikit-learn team — for TimeSeriesSplit and evaluation utilities

---

Made with ❤️ for Azamgarh, UP, India
⭐ If this project helped you, please give it a star!