🤖 AI Research Radar

This project is a complete, event-driven data pipeline that automatically tracks new AI research papers from arXiv, enriches them using an AI (Ollama) embedding model, and stores them in a queryable PostgreSQL database.

The entire system is containerized with Docker and orchestrated with Apache Airflow.

Future Plan

• The topic will be given by the users.
• New paper will be discovered.

---

🏛️ Architecture

This project is built as a decoupled, event-driven system using Kafka as a message bus. This separates concerns, making the system resilient and scalable.

1. Airflow arxiv_producer_dag: Runs on a schedule (e.g., hourly) and queries the arXiv API for new papers matching our criteria (e.g., cat:cs.AI).
2. Producer (Python): Publishes the metadata for each new paper as a JSON message into the raw_papers Kafka topic.
3. ML Consumer (ml_consumer.py):
   • Reads new messages from the raw_papers topic in batches.
   • Makes an API call to a local Ollama service (running nomic-embed-text) to get a 768-dimension vector embedding for each paper's abstract.
   • Calculates cosine similarity scores against a predefined list of user topics (e.g., "AI Agents").
   • Publishes a new, enriched JSON message (containing the original data + the embedding + the scores) to the analyzed_papers topic.
4. Sink Consumer (consumer-sink.py):
   • Reads from the analyzed_papers topic.
   • Parses the enriched JSON and writes the data to four separate tables in PostgreSQL in a single atomic transaction.
5. Airflow (Monitoring):
   • pipeline_health_check_dag: A sensor runs every 15 minutes to ensure new papers are arriving in the database.
   • db_cleanup_dag: Runs daily to delete data older than 90 days.

---

🛠️ Tech Stack

• Orchestration: Apache Airflow
• Message Bus: Apache Kafka
• Database: PostgreSQL + pgvector (for vector similarity search)
• AI/ML: Ollama (nomic-embed-text)
• Producers/Consumers: Python (kafka-python, ollama, psycopg2)
• Containerization: Docker & Docker Compose
• Testing: pytest & pytest-mock

---

🚀 How to Run

Prerequisites

1. Docker Desktop installed and running.
2. Ollama installed and running on your host machine.
3. Pull the nomic-embed-text model:

   ollama pull nomic-embed-text

1. Set Up the Environment

This project does not require any API keys, as it runs 100% locally.

# Clone the repository
git clone [https://github.com/YOUR_USERNAME/ai_research_radar.git](https://github.com/YOUR_USERNAME/ai_research_radar.git)
cd ai_research_radar

2. Build and Start All Services

This command will build the custom Airflow and Python images, then start all services (Kafka, Postgres, Airflow, and our two consumers) in the background.

docker-compose up -d --build

3. Start the Pipeline

The pipeline is now running, but no data will be produced until the Airflow DAG runs.

1. Open the Airflow UI at http://localhost:8080 (login: admin / admin).
2. In the Admin -> Connections tab, create a new connection:
   • Conn Id: ai_radar_db
   • Conn Type: Postgres
   • Host: postgres
   • Schema: ai_radar
   • Login: admin
   • Password: password
   • Port: 5432
3. On the main "DAGs" page, un-pause (toggle) the arxiv_producer_dag.
4. To get your first batch of data, click the "Play" (▶️) button on the arxiv_producer_dag to trigger a manual run.

4. Query Your Data

You can now query the enriched data directly from the Postgres container.

# 1. Connect to psql
docker-compose exec postgres psql -U admin -d ai_radar

# 2. Run the "Money Shot" query
# See the most relevant papers for your topics:
SELECT 
    p.title, 
    t.name AS topic, 
    pr.relevance_score
FROM papers p
JOIN paper_relevance pr ON p.id = pr.paper_id
JOIN topics t ON pr.topic_id = t.id
ORDER BY pr.relevance_score DESC
LIMIT 10;

5. Run Tests

To run the unit tests, use this command from the project root:

docker-compose run --rm consumer-sink pytest