Split per-state legislation repos by session for selective downloads

[sartaj]

Goal

Let users govbot pull just the sessions they care about, instead of grabbing every session a state has ever had. Today the per-state repos are small — but they only carry the current session. Once we backfill history (plus extract full bill text — #31), a single state repo balloons.

Where the data lives today

• Published to: github.com/govbot-data — 55 repos, one per US state/territory + federal ({xx}-legislation + usa-legislation). Each one looks like:

  country:us/state:ca/sessions/<session-id>/bills/<bill-id>/metadata.json
  

• How it gets there: the scraper in this repo at actions/scrape/action.yml (lines ~333–405) does git commit && git push to that state's repo when called with commit-and-push: true. The README inside each published repo (e.g. ca-legislation README) is the user-facing entry point.
• Current scale snapshot (May 2026, single session each): CA 53 MB, IL 24 MB, NY 22 MB, HI 19 MB, MA 18 MB, MO 18 MB, MT 18 MB. Many states <2 MB. With multi-year history × bill text extraction (Extract full bill text from PDFs for RAG #31), expect 10–100× growth.

Stale pointer in our registry

actions/govbot/data/registry.json still points every git_url at the old chn-openstates-files org (the predecessor of govbot-data). The govbot-data org has the same repo names and current data. We should migrate the registry to govbot-data as part of (or before) this work — otherwise users following AGENT.md land on a stale org.

Proposed: session-level partitioning

The shape the user proposed: a state's core repo (e.g. govbot-data/ca-legislation) becomes a thin index whose sessions/ entries are git submodules, each pointing at a per-session repo (e.g. govbot-data/ca-legislation-2025-2026). govbot pull ca only fetches the index + a chosen session's submodule; older sessions stay un-cloned until asked for.

Alternatives worth comparing before committing

The submodule path is clear conceptually but creates N states × M sessions = potentially 500+ repos to manage in the org, plus submodule UX is famously rough (detached HEADs, git submodule update --init --recursive, etc.). Two lighter options to weigh:

1. Sparse-checkout + partial clone. Keep one repo per state. govbot pull --session 2025-2026 ca does git clone --filter=blob:none --sparse <repo> then git sparse-checkout set country:us/state:ca/sessions/2025-2026. Fetches only the blobs for the requested session. Zero new repos, zero submodule machinery. Probably the simplest answer if the only requirement is "don't download all sessions."
2. Per-session repos referenced directly by registry. Skip the parent repo entirely. registry.json gains entries like us-legislation/ca/2025-2026 → govbot-data/ca-legislation-2025-2026. Selection happens by registry key, no submodules anywhere. Cleanest model, biggest scraper-side change.

Submodules give you "one URL to clone everything" for users who do want full history; sparse-checkout gives you the same selectivity with simpler ops; per-session-repos give you the cleanest publishing model.

What this issue should produce

• A decision between the three options above (submodules vs. sparse-checkout vs. per-session repos in the registry).
• Updates to actions/scrape/action.yml if the publishing layout changes (submodule wiring after push, or new per-session-repo creation).
• Updates to govbot pull in actions/govbot/src/main.rs (perform_clone_operations, lines ~536–612) to support --session selection.
• Migration of registry.json from chn-openstates-files → govbot-data regardless of which option wins.
• A line in AGENT.md and the per-state repo READMEs explaining how to fetch only specific sessions.

Acceptance

• govbot pull ca --session 2025-2026 fetches noticeably less data than govbot pull ca (verify with du -sh ~/.govbot/cache/).
• govbot doctor validates the chosen layout works for a fresh clone.
• The govbot-data org has a clear "pick your session" path documented in at least one state's README.

References

• Org: https://github.com/govbot-data
• Sample state repo: https://github.com/govbot-data/ca-legislation
• Registry: actions/govbot/data/registry.json
• Scraper push: actions/scrape/action.yml (commit-and-push block)
• Pull logic: actions/govbot/src/main.rs → perform_clone_operations()
• Cache layout: actions/govbot/src/cache.rs
• Related: Extract full bill text from PDFs for RAG #31 (full bill text extraction — main driver of future size growth)