Add Superset asset management system with automated sync

[blarghmatey]

Overview

Implements complete asset management workflow for Apache Superset using the sup CLI with OAuth authentication and automated database UUID mapping.

New Features

• Automated export/import of all Superset assets (datasets, charts, dashboards)
• OAuth2 PKCE authentication with CSRF token support for self-hosted Superset
• Automatic database UUID mapping between environments
• Pagination support to fetch ALL assets (not limited to 50)
• Environment-agnostic scripts (works with any source/target instance)
• Continue-on-error for resilient imports

Scripts Added

• export_all.sh: Export all assets from any Superset instance with pagination
• sync_assets.sh: Sync assets between instances with UUID mapping
• map_database_uuids.py: Automatic database UUID translation
• validate_assets.sh: YAML validation (from previous workflow)
• promote_to_production.sh: Legacy promotion script (from previous workflow)
• export_from_qa.sh: Legacy QA export script (from previous workflow)

Assets Included

• 76+ datasets from production Trino and Superset Metadata DB
• 107+ charts covering all visualization types
• 18+ published dashboards (enrollment, engagement, orders, etc.)
• 2 database connection configs (Trino + Superset Metadata DB)

Documentation

• WORKFLOWS.md: Complete workflow guide with diagrams
• scripts/README.md: Script reference and troubleshooting
• Both include examples for common operations

Technical Implementation

• Uses sup CLI (fork: mitodl/superset-sup with self-hosted support)
• CSRF token handling for POST requests (multipart/form-data)
• Database UUID mapping: Production Trino → QA Trino translation
• Pagination loops fetch 100 items per page until complete
• Regex fallback for malformed JSON from CLI

Typical Workflows

1. Production → QA sync (backup/mirroring)
2. QA → Production promotion (after testing changes)
3. Weekly backups with git version control

Related Work

• Requires sup CLI from https://github.com/mitodl/superset-sup/tree/self_hosted_superset_oidc
  • Install with uv tool install git+https://github.com/mitodl/superset-sup@self_hosted_superset_oidc
• See PR feat: Complete self-hosted Superset support with OAuth2/OIDC and asset management preset-io/superset-sup#19 for upstream sup CLI improvements
• CSRF token and pagination fixes already merged in our fork

Testing

✅ Tested and validated:

• Full export from production (76 datasets, 107 charts, 18 dashboards)
• Sync to QA with automatic UUID mapping
• All assets successfully imported to QA
• OAuth authentication with CSRF tokens working
• Pagination fetches complete asset list (no 50-item limit)
• Continue-on-error handles problematic assets gracefully

To run it yourself, run uv tool install git+https://github.com/mitodl/superset-sup@self_hosted_superset_oidc and then create a configuration file located at ~/.sup.config.yml with the following contents:

current_instance_name: superset-production
superset_instances:
  superset-production:
    auth_method: oauth
    oauth_authorization_url: https://sso.ol.mit.edu/realms/ol-data-platform/protocol/openid-connect/auth
    oauth_client_id: ol-superset-cli
    oauth_scope: openid profile email
    oauth_token_url: https://sso.ol.mit.edu/realms/ol-data-platform/protocol/openid-connect/token
    url: https://bi.ol.mit.edu
  superset-qa:
    auth_method: oauth
    oauth_authorization_url: https://sso-qa.ol.mit.edu/realms/ol-data-platform/protocol/openid-connect/auth
    oauth_client_id: ol-superset-cli
    oauth_scope: openid profile email
    oauth_token_url: https://sso-qa.ol.mit.edu/realms/ol-data-platform/protocol/openid-connect/token
    url: https://bi-qa.ol.mit.edu

[rachellougee]

Nice work! I wasn't able to run the export script locally but I can circle back next week.