Big Spatial Data Processing with Apache Sedona – NYC Taxi Hotspot Analysis

Project Overview

This project demonstrates large-scale geospatial data processing using Apache Spark and Apache Sedona.
The pipeline processes NYC Taxi trip records, integrates spatial taxi zone polygons, and identifies high-demand pickup hotspots across New York City.

The goal is to showcase distributed spatial data processing, transforming millions of trip records into geographic insights about urban mobility patterns.

Objectives

• Process large-scale taxi trip datasets using Apache Spark
• Integrate spatial polygon datasets using Apache Sedona
• Identify taxi pickup hotspots across NYC zones
• Analyze temporal demand patterns by hour
• Produce optimized Parquet analytics datasets
• Visualize taxi demand using an interactive geospatial map

System Architecture

flowchart TD

A[NYC TLC Taxi Dataset] --> B[Apache Spark Processing]
B --> C[Data Cleaning & Bronze Layer]
C --> D[Apache Sedona Spatial Processing]
D --> E[Taxi Zone Spatial Join]
E --> F[Hotspot Aggregation]
F --> G[Partitioned Parquet Storage]
G --> H[Geospatial Visualization]

Loading

Data Sources

• NYC Taxi & Limousine Commission (TLC) Trip Records
  https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

• NYC Taxi Zone Polygons
  Official shapefile containing taxi zone boundaries (263 zones)

Dataset characteristics:

Dataset	Description
Taxi trip records	~3 million trips (Jan 2025)
Taxi zones	263 spatial polygons
Trip data format	Parquet
Spatial format	Shapefile

Methods

1. Load NYC taxi trip records using Apache Spark
2. Create a cleaned bronze dataset with relevant columns
3. Load taxi zone polygons using Apache Sedona
4. Convert zone polygons to spatial geometries
5. Join trip pickup locations with taxi zones
6. Perform spatial aggregation to identify pickup hotspots
7. Analyze pickup demand patterns by hour
8. Store analytics outputs in optimized Parquet datasets
9. Visualize taxi demand intensity on an interactive NYC map

Key Metrics

• Total trips analyzed: ~3,000,000
• Taxi zones analyzed: 263
• Top pickup zone: Midtown Center (Manhattan)
• Peak pickup hours: 17:00 – 19:00

Example results:

Borough	Zone	Pickup Count
Manhattan	Midtown Center	169,977
Manhattan	Upper East Side South	163,703
Manhattan	Upper East Side North	155,647
Queens	JFK Airport	146,137

Key Insights

• Manhattan dominates taxi pickup demand, particularly Midtown areas.
• Airports such as JFK and LaGuardia represent major transportation hubs with high taxi activity.
• Taxi demand peaks during evening commute hours (5–7 PM).
• High-demand zones align closely with commercial and tourism districts.

These insights demonstrate how spatial analytics can support urban mobility analysis and transportation planning.

Visualization

Taxi demand hotspots visualized across NYC zones.

The visualization highlights demand concentration in:

• Midtown Manhattan
• Upper East Side
• Times Square / Theatre District
• JFK Airport

Pipeline Architecture

Raw Taxi Data (TLC)
↓
Spark Data Processing (PySpark)
↓
Spatial Processing (Apache Sedona)
↓
Taxi Zone Join
↓
Hotspot Aggregation
↓
Partitioned Parquet Output
↓
Geospatial Visualization (Folium)

Output Datasets

The pipeline produces analytics-ready Parquet datasets.

data/gold/
├── pickup_hotspots/
│ └── yellow_2025_01
│
├── pickup_hotspots_by_hour/
│ └── yellow_2025_01

These datasets can be used for:

• mobility analytics
• demand forecasting
• urban transportation studies
• geospatial dashboards

Tools & Technologies

• Apache Spark
• Apache Sedona
• PySpark
• Docker
• Parquet
• Folium
• Python

Key capabilities demonstrated:

• distributed data processing
• geospatial analytics at scale
• spatial joins and aggregations
• optimized data pipeline design
• interactive geospatial visualization

Project Structure

spatial-bigdata-sedona
│
├── src
│ └── sedona_processing.py
│
├── data
│ ├── raw
│ ├── bronze
│ ├── gold
│ └── spatial
│
├── parquet_output
│
├── images
│ └── nyc_pickup_heat_2025_01.png
│
├── README.md
└── .gitignore

How to Run

Start the Apache Sedona container: docker run -p 8888:8888 -p 4040:4040 -v C:/workspace:/workspace apache/sedona:latest

Then run the Jupyter notebook pipeline to process the dataset.

Skills Demonstrated

This project highlights several key data engineering and analytics skills:

• Big Data Processing (Apache Spark)
• Geospatial Analytics (Apache Sedona)
• Data Pipeline Architecture
• Distributed Spatial Joins
• Data Engineering Best Practices
• Interactive Map Visualization

---

o## Key Takeaways

This project demonstrates how distributed spatial analytics can uncover meaningful mobility patterns from large transportation datasets.

Key findings include:

• Manhattan dominates taxi pickup demand, particularly in Midtown and Upper East Side zones.
• Airport zones (JFK and LaGuardia) represent major transportation hubs with consistently high taxi activity.
• Taxi demand peaks during evening commute hours (17:00–19:00).
• Spatial aggregation reveals demand clusters that align with commercial, tourism, and transit areas.

These insights highlight how geospatial analytics can support:

• urban mobility planning
• transportation infrastructure optimization
• demand forecasting for ride-hailing services.

---

Advanced Spatial Analysis

Beyond simple joins, this project demonstrates spatial operations using Apache Sedona.

Example spatial query:

• Taxi zone polygons loaded as geometries
• Spatial transformations applied to convert coordinate systems
• Zone centroids generated for heatmap visualization

Example spatial transformation: ST_Transform(geometry, 'epsg:2263', 'epsg:4326')

This converts taxi zone geometries from a projected coordinate system into WGS84 geographic coordinates suitable for web mapping.

Spatial centroids were then used to create a demand intensity heatmap.

---

Example Output (Top Pickup Zones)

Rank	Borough	Zone	Pickup Count
1	Manhattan	Midtown Center	169,977
2	Manhattan	Upper East Side South	163,703
3	Manhattan	Upper East Side North	155,647
4	Queens	JFK Airport	146,137
5	Manhattan	Times Sq / Theatre District	125,829

These zones correspond to major commercial districts and transportation hubs.

---

Demand Visualization

The spatial demand distribution can be visualized using an interactive heatmap.

Example visualization:

---

Performance Considerations

Because the dataset contains millions of taxi trips, several optimizations were applied:

• Column pruning to reduce memory usage
• Distributed aggregation using Spark
• Partitioned Parquet outputs
• Repartitioning by pickup_hour for efficient analytics queries

These practices mirror real-world data engineering pipelines used in production systems.

---

Future Improvements

Potential extensions of this project include:

• Spatial clustering of taxi demand hotspots
• Demand forecasting using machine learning
• Integration with real-time mobility data streams
• Interactive dashboards using Apache Superset or Streamlit
• Multi-month or yearly trend analysis

---

Author

Pattarin Thunyapar

Master’s Degree – Data Analytics
Berlin School of Business and Innovation

Technical interests:

• Data Engineering
• Geospatial Analytics
• Machine Learning
• Distributed Systems

---