A proof of concept for collecting, processing, and analyzing news articles using AI and modern data technologies. Built on top of PyArrow Acero for streaming data processing, Delta Lake for efficient data storage, and OpenAI's GPT models for advanced text analysis.
- Real-time News Collection: Automated RSS feed collection from BBC News with extensible architecture
- AI-Powered Analysis: Named entity extraction using OpenAI's GPT models (actor, role, category)
- Efficient data management: PyArrow Acero for high-performance streaming data processing and Delta Lake for ACID-compliant data storage and time travel
- Interactive Dashboard: Streamlit-based visualization of news data and actor analysis
A more detailed explanation of the code can be found here.
# Create and activate virtual environment
python -m venv venv
source venv/bin/activate
# Install dependencies
pip install -r requirements.txtexport OPENAI_API_KEY="your-api-key"
export OPENAI_ORG_ID="your-org-id" # Optional
export OPENAI_PROJECT="your-project" # Optional# Collect and process news data
python main.py
# Launch the interactive dashboard
streamlit run visualizer.pyThe ETL keeps track of the processed RSS entries in the JSON file ./rss_state.json. If you want to reprocess them RSS Feeds, delete the file.
The Streamlit dashboard provides an interactive interface to explore the collected data:
- News article timeline
- Actor analysis with role distribution
- Category distribution
- Detailed article view with actor information
The processed data is stored in Delta Lake tables, making it easy to analyze using your favorite data science tools:
from deltalake import DeltaTable
# Load actor data
dt_actors = DeltaTable("/tmp/bbc_news/curated/actors")
df_actors = dt_actors.to_pandas()
df_actors.head()Will give you:
news_id actor_name actor_role is_main_actor
0 https://www.bbc.com/news/articles/cp913ze3k9jo#0 Microsoft company shutting down Skype True
1 https://www.bbc.com/news/articles/crkx3vy54nzo#0 Co-op affected company True
2 https://www.bbc.com/news/articles/crkx3vy54nzo#0 BBC contacted by hackers False
3 https://www.bbc.com/news/articles/c9856ge2742o#0 Co-op retailer True
4 https://www.bbc.com/news/articles/c86jx18y9e2o#0 Apple technology giant True- News Collection: The
NewsCollectorclass fetches news articles from BBC RSS feeds - Data Processing: Uses PyArrow Acero for efficient data processing
- AI Analysis: The
NewsProcessorclass uses OpenAI's GPT models to:- Extract named entities (actors)
- Identify their roles
- Classify events
- Storage: Results are stored in Delta Lake tables for efficient querying
- Visualization: Streamlit dashboard for interactive data exploration
Even if it is a proof of concept, there are several areas for improvement:
- High cardinality of roles: The model needs refinement regarding categories, which are not used right now, or roles.
- Process data in a streaming fashion: Acero is a streaming engine but requires an orchestrator to keep the tasks running in a loop, and to monitor them (e.g. Dagster, Prefect, Airflow, ...)
- RSS State: We keep track of the processed entities in a JSON file. A proper database with fast lookups should be used.
- Optimize entity extraction: The entities are extracted sequentially, with several transformations (OpenAI JSON -> PyDantic -> List -> PyArrow Table). We can make concurrent calls to OpenAI and reduce the transformations.
- Acero Documentation
- Acero GitHub Repository
- Delta Lake Documentation
- OpenAI API Documentation
- BBC News RSS Feeds
- Streamlit Documentation
This project is licensed under the Apache 2.0 License - see the LICENSE file for details.