Module Mirrorer

Mirrors configured repositories to the dune-mirrors org.

Mirrors last updated at: never

Configuration

The repositories to be mirrored are defined in repos.json. This file contains a JSON object mapping repository names to their GitLab URLs. To add or remove repositories from the mirroring process, simply edit this file and push your changes to trigger an update.

Example format:

{
    "repository-name": "https://gitlab.dune-project.org/path/to/repository.git"
}

Workflow Description

The mirroring process is implemented as a GitHub Actions workflow defined in .github/workflows/mirrorer.yml. The workflow operates as follows:

1. Trigger: The workflow runs:

   • Automatically every day at midnight (via cron schedule)
   • Manually when triggered via GitHub's workflow_dispatch
   • On push to the repository (e.g., when updating repos.json)

2. Repository Matrix Generation:

   • The workflow first runs a job that executes repos_to_matrix.py
   • This script reads repos.json and converts it to a format suitable for GitHub Actions' matrix strategy
   • Each repository is processed with its name, URL, and a keyname (used for SSH key reference)

3. Mirroring Process:

   • For each repository in the matrix, a separate job runs in parallel
   • The job mints a short-lived installation token from the DUNE Mirrorer GitHub App (its private key is stored in the DUNE_MIRRORER_PRIVATE_KEY secret), scoped to just the target mirror repository
   • It clones the GitLab repository with --mirror option to get all branches and tags
   • Sets the push URL to the corresponding repository in the dune-mirrors GitHub organization
   • Removes the GitLab merge-request references (which GitHub rejects on push)
   • Pushes all branches and tags to GitHub with --mirror option, authenticating with the App token

Authentication

Pushes to the dune-mirrors org are authenticated with the DUNE Mirrorer GitHub App rather than per-repository SSH deploy keys. The App must be installed on the organization with Contents: write permission and granted access to the mirror repositories. Only its private key (the DUNE_MIRRORER_PRIVATE_KEY Actions secret) needs to be configured.

Installation and Dependencies

This project uses pyproject.toml for dependency management. The required dependencies are:

• requests: For making HTTP requests to the GitHub API
• python-dotenv: For loading environment variables from .env files

Installing Dependencies

To install the dependencies, you can use pip:

# Install the base dependencies
pip install -e .

# For development (includes testing and linting tools)
pip install -e ".[dev]"

Usage

• Add/Remove Repositories: Edit repos.json and push changes
• Manual Trigger: Use the "Run workflow" button in the Actions tab of the GitHub repository
• Automatic Updates: The workflow runs daily to ensure mirrors stay up-to-date

Local Development

To run the scripts locally:

1. Create a .env file with your GitHub token:

   GITHUB_TOKEN=your_github_token
   

2. Run the infrastructure setup script (ensures each mirror repo exists in the org; authentication for pushes is handled by the DUNE Mirrorer GitHub App):

   python infra_setup.py

3. Generate the repository matrix:

   python repos_to_matrix.py