Deprecate 'FCAS Providers' static table — AEMO migrated the data (#92)#93
Merged
Conversation
AEMO has emptied the `Ancillary Services` sheet of `NEM Registration and Exemption List.xlsx` and migrated the data to a weekly archive at https://www.nemweb.com.au/REPORTS/CURRENT/ANCILLARY_SERVICES_REPORTS/ (file pattern: `PUBLIC_ANCILLARY_SERVICES_YYYYMMDD.zip`). The current NEMOSIS handler downloads the workbook fine but parses the now-empty sheet and chokes on the `drop_duplicates(['Bid Type', 'DUID'])` step because those columns no longer exist anywhere in the workbook. `select_columns='all'` doesn't help — the dedup runs unconditionally. This is a fresh problem, not the original #69. #69 was about a missing entry in `static_downloader_map`, with a workaround of priming the cache via the sibling `Generators and Scheduled Loads` table. That workaround still gets the file on disk but no longer helps because the sheet's contents are gone at the AEMO source. Closing #69 in favour of the new tracking issue (#92). Changes: 1. `static_table` now early-raises `UserInputError` for `FCAS Providers` with a message that names the new endpoint and references #92, so users have somewhere to go. Raised before any other validation so the message is consistent regardless of cache state or other args. 2. `defaults.static_tables` drops the `FCAS Providers` entry so it doesn't appear in the user-iterable list of supported static tables. The other metadata dicts (`names`, `table_types`, `table_columns`, etc.) keep their entries — they're internal, and removing them widens the diff without changing the user-facing contract. 3. `tests/end_to_end_table_tests/test_static_tables.py` replaces the two FCAS-Providers happy-path tests with two deprecation-error tests: one asserts the exception fires, one locks the new endpoint URL and the issue number into the message so doc-drift in either direction breaks the test. 4. README's `print(defaults.static_tables)` doctest output drops the FCAS Providers entry, with a follow-up note explaining where the data moved and linking to #92. The longer-term fix is to rewrite the handler against the new weekly archive endpoint. That requires schema discovery (column names, types, primary keys all need to be re-derived from a current zip) and is parked under #92 pending interest. Closes #69. Tracks under #92. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ancillary Servicessheet ofNEM Registration and Exemption List.xlsx(the workbook NEMOSIS reads forFCAS Providers) and migrated the data to a weekly archive at https://www.nemweb.com.au/REPORTS/CURRENT/ANCILLARY_SERVICES_REPORTS/. The existing handler downloads the workbook, finds the sheet empty, and chokes on thedrop_duplicates(['Bid Type', 'DUID'])step because those columns no longer exist anywhere in the file.select_columns='all'doesn't sidestep it — the dedup runs unconditionally.static_table('FCAS Providers', ...)now raises a clearUserInputErrorimmediately, with a message naming the new endpoint and linking to AEMO migrated FCAS Providers data out of the registration XLS; current handler returns nothing #92 for tracking.FCAS Providersfromdefaults.static_tablesso it no longer appears in the user-iterable list of supported tables. Other metadata dicts keep their entries (smaller diff, no user-visible difference).print(defaults.static_tables)doctest output updated, with a follow-up note explaining the migration and pointing at AEMO migrated FCAS Providers data out of the registration XLS; current handler returns nothing #92.Relationship to #69
#69 was about a missing entry in
static_downloader_mapcausingNoDataToReturnon a cold cache forFCAS Providers. The workaround at the time was to prime the cache via the siblingGenerators and Scheduled Loadstable (same workbook). That workaround still gets the file on disk but no longer helps because the sheet's contents are gone at the AEMO source — making #69 effectively obsolete. Closing it in favour of #92.Why deprecate now vs. rewrite
A rewrite against the new endpoint is the right long-term answer but requires schema discovery from scratch — the new
PUBLIC_ANCILLARY_SERVICES_YYYYMMDD.zipfiles have different columns, types, and (probably) primary keys than the old sheet. Deprecation gets users a clear error message and a forwarding address today; the rewrite is parked under #92 pending contributor interest.Test plan
uv run pytest tests/ignoring the network-hitting old suites).test_fcas_providers_raises_deprecation_errorconfirms the exception fires.test_fcas_providers_deprecation_message_points_to_new_endpointconfirms the message names the new endpoint and issue AEMO migrated FCAS Providers data out of the registration XLS; current handler returns nothing #92 — locks documentation against drift.Closes #69.
🤖 Generated with Claude Code