Skip to content

observability.md

Latest commit

 

History

History
270 lines (193 loc) · 11.2 KB

observability.md

File metadata and controls

270 lines (193 loc) · 11.2 KB
Raw

OpenTelemetry Observability Stack

This guide explains how to use the OpenTelemetry (OTEL) observability stack included with the MADSci example lab.

Overview

The MADSci example lab includes a complete observability stack that provides:

  • Traces: Distributed tracing to understand request flow across services
  • Metrics: Performance and operational metrics from all managers
  • Logs: Centralized log aggregation with trace correlation

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                        MADSci Example Lab                            │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐               │
│  │ EventManager │  │ WorkcellMgr  │  │ DataManager  │  ...          │
│  │ (traces)     │  │ (traces)     │  │ (traces)     │               │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘               │
│         │                  │                  │                      │
│         └──────────────────┼──────────────────┘                      │
│                            │ OTLP (gRPC :4317)                       │
│                            ▼                                         │
│  ┌──────────────────────────────────────────────────────────────────┐│
│  │                    OTEL Collector                                ││
│  │  Receives → Processes → Exports to backends                     ││
│  └──────────────────────────────────────────────────────────────────┘│
│                            │                                         │
│         ┌──────────────────┼──────────────────┐                      │
│         ▼                  ▼                  ▼                      │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐               │
│  │    Jaeger    │  │  Prometheus  │  │    Loki      │               │
│  │  (Traces)    │  │  (Metrics)   │  │   (Logs)     │               │
│  │  :16686      │  │  :9090       │  │  :3100       │               │
│  └──────────────┘  └──────────────┘  └──────────────┘               │
│                            │                                         │
│                            ▼                                         │
│  ┌──────────────────────────────────────────────────────────────────┐│
│  │                      Grafana                                     ││
│  │           Unified Dashboard (Traces + Metrics + Logs)            ││
│  │                        :3000                                     ││
│  └──────────────────────────────────────────────────────────────────┘│
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Quick Start

Basic Mode (Collector + Debug Output)

The base compose.yaml includes the OTEL collector with debug output. Traces and metrics are received but only logged to the collector's stdout.

docker compose up

Full Observability Stack

To enable the full observability stack with Jaeger, Prometheus, Loki, and Grafana:

docker compose --profile otel up

Accessing the UIs

All services use host network mode:

Service URL Description
Grafana http://localhost:3000 Unified dashboards (admin/admin)
Jaeger http://localhost:16686 Distributed tracing UI
Prometheus http://localhost:9090 Metrics querying
Loki http://localhost:3100 Log aggregation API

Note: Jaeger's OTLP receiver is configured on ports 14317 (gRPC) and 14318 (HTTP) to avoid conflicts with the OTEL collector's receiver ports (4317/4318).

Default Credentials

  • Grafana: admin / admin (you'll be prompted to change on first login)

To set a different Grafana password, set the environment variable before starting:

export GRAFANA_ADMIN_PASSWORD=your-secure-password
docker compose --profile otel up

What's Included

Pre-configured Grafana

  • Datasources: Jaeger, Prometheus, and Loki are automatically configured
  • Dashboards: MADSci Lab Overview dashboard is pre-installed
  • Correlations: Trace IDs in logs link directly to Jaeger traces

OTEL Collector Pipelines

  1. Traces Pipeline: OTLP → Batch Processing → Jaeger
  2. Metrics Pipeline: OTLP → Batch Processing → Prometheus Remote Write
  3. Logs Pipeline: OTLP → Batch Processing → Loki

Configuration

Enabling OTEL in Managers

OTEL is enabled per-manager via environment variables. The example lab's .env file has these enabled by default:

# Event Manager
EVENT_OTEL_ENABLED=true
EVENT_OTEL_SERVICE_NAME="madsci.event"
EVENT_OTEL_EXPORTER="otlp"
EVENT_OTEL_ENDPOINT="http://localhost:4317"
EVENT_OTEL_PROTOCOL="grpc"

# Similar for other managers...

Disabling OTEL

To disable OTEL for a specific manager, set:

EVENT_OTEL_ENABLED=false

Custom Configuration

Configuration files are located in examples/example_lab/otel/:

  • otel-collector-full.yaml: Full collector config with all exporters
  • prometheus.yaml: Prometheus configuration
  • loki.yaml: Loki configuration
  • grafana/provisioning/: Grafana auto-provisioning configs

Viewing Traces

In Jaeger

  1. Open http://localhost:16686
  2. Select a service from the dropdown (e.g., madsci.event)
  3. Click "Find Traces"
  4. Click on a trace to see the full request flow

In Grafana

  1. Open http://localhost:3000
  2. Navigate to "Explore"
  3. Select "Jaeger" datasource
  4. Search for traces by service or trace ID

Viewing Metrics

In Prometheus

  1. Open http://localhost:9090
  2. Use PromQL queries like:
    • sum(rate(otelcol_receiver_accepted_spans[5m])) - Spans per second
    • sum(rate(otelcol_receiver_accepted_metric_points[5m])) - Metrics per second

In Grafana

  1. Open http://localhost:3000
  2. Go to the "MADSci Lab Overview" dashboard
  3. View telemetry rates and trends

Viewing Logs

In Grafana

  1. Open http://localhost:3000
  2. Navigate to "Explore"
  3. Select "Loki" datasource
  4. Use LogQL queries like:
    • {job=~".+"} - All logs
    • {service_name="madsci.event"} - Logs from Event Manager

Log Bridge

When OTEL is enabled, MADSci automatically bridges Python's standard logging module to the OTEL log pipeline using LoggingInstrumentor from the opentelemetry-instrumentation-logging package. This means:

  • All stdlib log records are automatically exported to the OTEL collector
  • Trace context (trace_id, span_id) is injected into every log record
  • No manual log handler configuration is needed — the bridge is set up during OTEL bootstrap

This replaces the older LoggingHandler approach (deprecated in the OpenTelemetry SDK) and is included as a dependency of madsci_common.

Trace Correlation

When OTEL is enabled, MADSci automatically:

  1. Propagates trace context across HTTP requests between services
  2. Includes trace_id/span_id in structured log events (via the LoggingInstrumentor bridge)
  3. Links logs to traces via Grafana's derived fields

This allows you to:

  • Click on a trace ID in logs to jump to the full trace
  • See which logs were generated during a specific request
  • Understand the full flow of a workflow across all managers

Troubleshooting

No traces appearing in Jaeger

  1. Check that OTEL is enabled in the manager:

    docker compose logs event_manager | grep -i otel

  2. Verify the collector is receiving data:

    docker compose logs otel_collector | grep -i span

  3. Ensure the collector is running:

    docker compose ps otel_collector

Collector showing export errors

Check that all backend services are running:

docker compose --profile otel ps

Grafana can't connect to datasources

The datasources use localhost URLs which work with network_mode: host. If you're not using host networking, update the datasource URLs in grafana/provisioning/datasources/datasources.yaml.

Data Persistence

Telemetry data is stored in .madsci/ subdirectories:

  • .madsci/jaeger/ - Trace data
  • .madsci/prometheus/ - Metrics data
  • .madsci/loki/ - Log data
  • .madsci/grafana/ - Grafana settings and dashboards

To reset all observability data:

docker compose --profile otel down
rm -rf .madsci/jaeger .madsci/prometheus .madsci/loki .madsci/grafana

Production Considerations

This observability stack is designed for development and demonstration. For production:

  1. Security: Use proper authentication, don't expose UIs publicly without protection
  2. Storage: Configure appropriate retention periods and storage backends
  3. Sampling: Enable trace sampling to reduce data volume
  4. High Availability: Consider distributed deployments for Jaeger, Prometheus, and Loki
  5. Secrets: Use proper secrets management for credentials

Further Reading