Skip to content

Automate weekly cross-repo dependency alert sync - #195

Merged
ibrahim halatci (ihalatci) merged 11 commits into
masterfrom
automation/security-sync-orchestrator
Jul 1, 2026
Merged

Automate weekly cross-repo dependency alert sync#195
ibrahim halatci (ihalatci) merged 11 commits into
masterfrom
automation/security-sync-orchestrator

Conversation

@ihalatci

@ihalatci ibrahim halatci (ihalatci) commented Mar 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Implements weekly cross-repo dependency alert orchestration (Sunday 02:00 UTC) driven by packagingApp credentials. Manual approval/merge of the resulting PRs is preserved.

What this adds

Orchestrator workflow — .github/workflows/dependency-security-sync.yml

  • Runs weekly on schedule and on manual workflow_dispatch
  • Mints a packagingApp token via actions/create-github-app-token@v1 (GH_APP_ID / GH_APP_KEY org secrets)
  • Fetches open Dependabot alerts from citusdata/citus
  • Invokes the sync script (see below) to mutate citus Pipfiles and regenerate lockfiles
  • Opens/updates the citus sync PR as a Draft on branch automation/dependency-security-sync (--base main)
  • Regenerates the four the-process requirements.txt files using pipenv requirements against the updated lockfile, with headers pointing at citusdata/citus#<CITUS_PR> (matching existing convention)
  • Opens/updates the the-process sync PR on the same branch (--base master)
  • Closes only those Dependabot PRs whose title bumps a package the run actually addressed (search: author:app/dependabot in:title Bump <pkg>); leaves unrelated Dependabot PRs alone
  • Surfaces a per-alert outcome table in the run summary

Sync script — .github/scripts/security_sync.py

CLI: --alerts <json> --citus-root <path> --summary-out <path>

Behavior:

  • Deduplicates alerts by package, keeps the highest first_patched_version per package
  • For each target, edits both Pipfiles (citus src/test/regress and .devcontainer/src/test/regress):
    • Direct dep already in the file → rewrites the constraint to ">=<patched>"
    • Transitive dep → appends <package> = ">=<patched>" under [packages] or [dev-packages] based on alert scope (runtime vs development), so pipenv lock will pull a fixing version
  • Runs pipenv lock in both regress trees
  • Reads each post-state lockfile and classifies every target as applied / already-satisfied / not-fixed / absent, with an overall verdict of addressed / already-satisfied / failed
  • Writes sync-summary.json ({ "addressed": [...], "details": [...] }) consumed by the workflow
  • Exits non-zero when nothing was addressed, so the workflow fails loudly and does not open PRs nor close any Dependabot PRs

Post-merge callback — .github/workflows/dependency-security-post-merge.yml

  • On merge of the the-process sync PR to master, computes the new image postfix from the merged SHA (-v<sha7>)
  • Updates image_suffix in citus/.github/workflows/build_and_test.yml on the citus sync branch so the corresponding citus PR picks up rebuilt images

Design guarantees

  1. No fix → no PR. If alerts cannot be resolved (e.g., upstream dep caps the patched range, no actionable first_patched_version, or pipenv lock fails), the workflow fails and opens nothing. Dependabot PRs are not closed.
  2. Narrow PR closure. Dependabot PRs are closed only when the consolidated PR actually addresses that specific package.
  3. No package-specific hardcoding. Alert payload drives all updates.
  4. Header convention preserved. Regenerated requirements.txt headers reference citusdata/citus#<PR_NUMBER>.

Operational prerequisites

  • Org secrets available to both repos:
    • GH_APP_ID
    • GH_APP_KEY
  • packagingApp GitHub App installed on citusdata/citus and citusdata/the-process with at minimum:
    • Contents: read & write
    • Pull requests: read & write
    • Issues: read & write (for PR comments)
    • Dependabot alerts: read (on citus)
    • Workflows: write (post-merge step edits a workflow file in citus)
  • Automation branch in both repos: automation/dependency-security-sync

Known upstream blockers

At time of writing, the live pyOpenSSL Dependabot alerts on citus cannot be resolved by this automation because mitmproxy 12.2.2 (citus's git-pinned version) declares pyOpenSSL<=25.3.0,>=24.3 while the alert wants >=26.0.0. The workflow correctly fails and opens no PR until mitmproxy is bumped past that cap. This is a real upstream constraint, not an automation defect.

Smoke testing performed (local mirror in /tmp/smoke-test)

Scenario Expected Actual
Empty alerts exit 1, no PR ✅
Already-satisfied alert (werkzeug 3.1.1) exit 1, classified already-satisfied ✅
Unfixable transitive (pyopenssl 26 blocked by mitmproxy cap) exit 1, no PR ✅
Non-existent target version exit 1, no PR ✅

Live happy-path (addressed != []) will be exercised on the first scheduled run after merge once a real upstream patch is available.

Follow-ups (not in this PR)

  • Optionally suppress Dependabot's own PR opening on citusdata/citus after this lands (alerts-only mode) so the consolidated PR is the single channel.
  • Promote this PR from Draft → Ready for review once the App permissions above are confirmed.

@ihalatci ibrahim halatci (ihalatci) mentioned this pull request Mar 4, 2026
Merged
@ihalatci
Trigger CI re-run on automation branch
fb4e1b8
@ihalatci
Merge remote-tracking branch 'origin/master' into automation/security…
6a13b1a 
…-sync-orchestrator
@ihalatci
Use $'...' for newline in automated PR bodies
6e92108
@ihalatci
Use GH_APP_ID / GH_APP_KEY org secrets for App auth
cfd1fd2
@ihalatci
Remove accidentally committed empty .vscode/settings.json
d1a8d35
@ihalatci
security_sync: handle transitive deps, fail-fast, narrow PR closures
f6e7548 
- update_pipfile() appends to [packages]/[dev-packages] when alert is for a
  transitive dep, so pipenv lock can pull the patched version.
- Use >= constraint instead of == to avoid over-pinning.
- Verify per-alert outcome by reading both Pipfile.lock files post-lock,
  classify as addressed / already-satisfied / failed.
- Write summary JSON consumed by the workflow; print one line per alert.
- Exit non-zero (no PRs opened) when no alerts are actually addressed.
- Workflow: pass --summary-out, surface details in run summary, expose
  addressed-package list as step output, only close Dependabot PRs whose
  title matches a package we addressed (instead of closing all of them).
@ihalatci
Create citus dependency sync PR as draft
f94d8cb
@ihalatci
Reference citus PR number in regenerated requirements headers
d95c1be 
Defer requirements regeneration from the Python script to a workflow step
that runs after the citus PR is created, so the header can reference
`citusdata/citus#<PR_NUMBER>` (matching the existing convention) rather
than a transient SHA.
@ihalatci
ibrahim halatci (ihalatci) marked this pull request as ready for review June 18, 2026 07:59
@ihalatci-msft
Fix dependency-security-sync flaws and harden version bumping
ede255d 
Validation-driven fixes from a real 14-alert backlog pass plus earlier
review fixes for the weekly cross-repo Dependabot alert sync.

security_sync.py:
- Add bump_lower_bound(): raise only the lower bound, preserve upper-bound
  and exclusion clauses (<, <=, !=, ~=), keep exact-pin discipline
  (==46.0.7 -> ==48.0.1, never downgrade), and flag a hard conflict
  (patched excluded by an existing cap) as not-fixed instead of silently
  widening the range. Replaces the old whole-spec '>={patched}' clobber.
- Lock each regress tree once per distinct post-edit Pipfile hash and copy
  the resolved Pipfile.lock to the byte-identical twin; remember a failed
  resolve so the twin is also classified not-fixed.
- Green no-op when no alert has a first_patched_version; only exit non-zero
  when a package genuinely failed.

safe_push.sh (new): extract the force-push guard that skips when the remote
automation branch has non-bot commits.

workflows: GitHub App token auth (create-github-app-token@v2, GH_APP_ID /
GH_APP_KEY, owner citusdata) exported once to the environment; paginated
alert fetch; PR steps gated on a non-empty addressed list; post-merge closes
superseded Dependabot PRs only after the sync PR merges and waits for
build-test-images before bumping citus image_suffix.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@ihalatci ibrahim halatci (ihalatci) changed the title Draft: automate weekly cross-repo dependency alert sync Automate weekly cross-repo dependency alert sync Jun 30, 2026
@ihalatci
ibrahim halatci (ihalatci) merged commit 61169c3 into master Jul 1, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ihalatci @ihalatci-msft