AWS Real-Time ML Streaming Platform

A local-first real-time machine learning streaming platform that simulates event ingestion, validation, feature processing, inference, anomaly detection, routing, alerting, monitoring, and drift/evaluation readiness with AWS-oriented architecture mapping.

Project Status

Current milestone: Milestone 10: AWS Architecture Readiness and Portfolio Polish.

This repository is a complete local simulation. It does not connect to AWS, create AWS resources, deploy infrastructure, run Terraform/CDK, or use paid services. AWS services are referenced only as conceptual production mappings.

Why This Project Matters

Real-time ML systems are used when organizations need to act on events quickly: suspicious transactions, service degradation, unusual customer behavior, operational incidents, and model-quality shifts. This project demonstrates the engineering shape of such a system without requiring cloud spend or live infrastructure.

The result is a portfolio-ready project that shows how a production-style streaming ML platform can be designed, decomposed, tested, monitored, and mapped to AWS services.

Business Problem

Batch analytics can miss time-sensitive patterns. A business may need to identify risky transactions, abnormal user behavior, degraded services, or model-quality issues while events are still actionable.

This project simulates how a platform could process streaming events, score them, identify anomalies, route records for action, generate alerts, and produce monitoring/evaluation reports.

Technical Problem

The technical challenge is to design a modular real-time ML workflow that supports:

• Reproducible synthetic event generation.
• Event validation and invalid-record handling.
• Streaming-style feature processing.
• Low-latency inference simulation.
• Rule-based anomaly detection.
• Event routing and alert generation.
• Pipeline monitoring and reporting.
• Drift detection and evaluation readiness.
• Clear AWS conceptual mapping without provisioning infrastructure.

End-To-End Pipeline

1. Synthetic producers generate transaction, customer activity, and system health events.
2. Local stream validation separates valid and invalid records.
3. Feature processing converts events into model-ready feature records.
4. Local inference produces deterministic risk predictions.
5. Anomaly detection applies rule-based checks across features and predictions.
6. Routing assigns records to action-oriented queues.
7. Alert generation creates local alert records for actionable routes.
8. Monitoring summarizes pipeline health, latency, throughput, predictions, anomalies, routes, and alerts.
9. Drift/evaluation readiness builds baselines, detects drift, evaluates synthetic-label quality, and emits retraining-readiness signals.

Architecture Diagram

flowchart LR
    A["Synthetic Producers"] --> B["Local Stream Files"]
    B --> C["Validation"]
    C --> D["Feature Processing"]
    D --> E["Real-Time Inference Simulation"]
    E --> F["Anomaly Detection"]
    F --> G["Event Routing"]
    G --> H["Alert Generation"]
    H --> I["Monitoring Reports"]
    E --> I
    F --> I
    D --> J["Drift & Evaluation Readiness"]
    E --> J
    I --> J
    J --> K["Reports & Outputs"]

Loading

Local-First Design

Everything runs with local Python modules and local files. AWS credentials are not required. This makes the project easy to review, test, and extend before any future cloud deployment work.

AWS Service Mapping

Local module	Conceptual AWS mapping
producers	Amazon Kinesis producer / data stream
consumers	Kinesis consumer / Lambda-style validation
feature_processing	SageMaker Processing / EMR-style processing
inference	SageMaker real-time endpoint
anomaly_detection	SageMaker monitoring / CloudWatch anomaly logic
routing	EventBridge-style rules
alerts	CloudWatch alarm / SNS-style notification
monitoring	CloudWatch metrics
storage	S3 and DynamoDB-style persistence
drift_evaluation	SageMaker Model Monitor-style evaluation
CI	GitHub Actions

See docs/aws_architecture_mapping.md for the fuller mapping.

Folder Structure

config/                         Local YAML configuration
data/streaming/                 Generated, validated, and feature records
docs/                           Architecture, runbooks, and portfolio notes
outputs/                        Predictions, anomalies, alerts, metrics, signals
reports/                        Markdown summaries and evaluation reports
src/streaming_platform/         Source package
tests/                          Unit and integration-style tests
.github/workflows/              GitHub Actions CI

Milestone Roadmap

• Milestone 1: Repository foundation.
• Milestone 2: Synthetic streaming event generation.
• Milestone 3: Local stream consumption and validation.
• Milestone 4: Streaming feature processing.
• Milestone 5: Local real-time inference simulation.
• Milestone 6: Local anomaly detection.
• Milestone 7: Event routing and alert generation.
• Milestone 8: Monitoring and reporting.
• Milestone 9: Drift detection and evaluation readiness.
• Milestone 10: AWS architecture readiness and portfolio polish.

How To Run Locally

Create and activate a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run the entire local pipeline:

PYTHONPATH=src python3 -m streaming_platform.run_local_pipeline

For a smaller run:

PYTHONPATH=src python3 -m streaming_platform.run_local_pipeline --records 25

If installed as a package, the script entry point is:

streaming-platform-run-local

Full Pipeline Command Sequence

PYTHONPATH=src python3 -m streaming_platform.producers.generate_events
PYTHONPATH=src python3 -m streaming_platform.consumers.consume_stream
PYTHONPATH=src python3 -m streaming_platform.feature_processing.process_features
PYTHONPATH=src python3 -m streaming_platform.inference.run_inference
PYTHONPATH=src python3 -m streaming_platform.anomaly_detection.run_anomaly_detection
PYTHONPATH=src python3 -m streaming_platform.routing.run_routing_pipeline
PYTHONPATH=src python3 -m streaming_platform.monitoring.run_monitoring
PYTHONPATH=src python3 -m streaming_platform.monitoring.run_drift_evaluation

Outputs Generated

Key local outputs include:

• data/streaming/simulated_events.jsonl
• data/streaming/validated_events.jsonl
• data/streaming/feature_records.jsonl
• outputs/live_predictions.jsonl
• outputs/anomaly_events.jsonl
• outputs/routed_events.jsonl
• outputs/alert_events.jsonl
• outputs/pipeline_health_metrics.json
• outputs/baseline_metrics.json
• outputs/drift_metrics.json
• outputs/evaluation_metrics.json
• outputs/retraining_signals.json

Reports Generated

• reports/feature_processing_summary.md
• reports/inference_summary.md
• reports/anomaly_detection_summary.md
• reports/routing_alerts_summary.md
• reports/monitoring_summary.md
• reports/streaming_performance_report.md
• reports/drift_evaluation_report.md

Testing

Run all tests:

python3 -m pytest

The test suite covers event generation, validation, feature processing, inference, anomaly detection, routing, alerting, monitoring, drift detection, evaluation metrics, and the full local pipeline runner.

Portfolio Positioning

This project demonstrates the architecture and engineering practices behind a real-time ML platform without requiring cloud deployment. It is designed around production-style concerns: event-driven processing, data contracts, model-ready features, inference outputs, anomaly records, operational metrics, and evaluation artifacts.

The platform highlights:

• Real-time machine learning system design.
• Streaming event processing across heterogeneous event types.
• Event-driven architecture with explicit routing and alert outputs.
• Low-latency inference simulation through a deterministic local scorer.
• Rule-based anomaly detection with severity and contributing-field context.
• Operational monitoring for volume, quality, latency, throughput, routes, and alerts.
• Drift detection and evaluation readiness using local baselines and synthetic labels.
• MLOps production thinking around reproducibility, observability, data quality, and model-quality checks.
• AWS-aligned architecture design without provisioning AWS infrastructure.
• Local-first engineering with generated reports and outputs for auditability.

The project is intentionally honest about scope: it is a local simulation and architecture-ready foundation, not a deployed production system.

Skills Demonstrated

• Python package design with src/ layout.
• Real-time ML system decomposition.
• Streaming analytics simulation.
• Event validation and data quality checks.
• Feature engineering for heterogeneous events.
• Local inference contract design.
• Rule-based anomaly detection.
• Event routing and alert generation.
• Monitoring, reporting, and operational metrics.
• Drift detection and synthetic-label evaluation.
• AWS architecture mapping.
• CI-ready testing.

Documentation

• Portfolio summary
• AWS architecture mapping
• Local pipeline runbook
• Architecture diagrams
• Monitoring and reporting design
• Drift and evaluation design
• Future enhancements

Future Enhancements

Future work could add real AWS Kinesis integration, SageMaker endpoint deployment, SageMaker Model Monitor, EventBridge rules, SNS alerts, trained models, experiment tracking, load testing, dashboards, infrastructure-as-code, and CI/CD deployment workflows.

These are intentionally not implemented in the current version.

No AWS Resources Created

This repository is local-only. It does not require AWS credentials and does not create AWS resources.