diff --git a/README.md b/README.md index f5e2c6a..1bb9828 100644 --- a/README.md +++ b/README.md @@ -110,7 +110,7 @@ raw_data_cache = 'C:/Users/your_data_storage' price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache) ``` -Using the default settings of `dynamic_data_compiler` will download CSV data from AEMO's NEMWeb portal and save it to the `raw_data_cache` directory. It will also create a feather file version of each CSV (feather files have a faster read time). Subsequent `dynamic_data_compiler` calls will check if any data in `raw_data_cache` matches the query and loads it. This means that subsequent `dynamic_data_compiler` will be faster so long as the cached data is available. +Using the default settings of `dynamic_data_compiler` will download zip archives from AEMO's NEMWeb portal and save them to the `raw_data_cache` directory. It will then extract each CSV, create a parquet file version (parquet has excellent compression and good interop with downstream tools), and delete the extracted CSV. The downloaded zip is retained by default so that subsequent runs that need to rebuild the parquet (e.g. to add columns or switch format) do not have to re-download from AEMO. Subsequent `dynamic_data_compiler` calls will check if any data in `raw_data_cache` matches the query and load it from there. This means that subsequent `dynamic_data_compiler` calls will be faster so long as the cached data is available. See [Caching options](#caching-options) below to tune what stays on disk. A number of options are available to configure filtering (i.e. what data NEMOSIS returns as a pandas DataFrame) and caching. @@ -162,36 +162,60 @@ unit_dispatch_data = dynamic_data_compiler(start_time, end_time, 'DISPATCHLOAD', ###### Caching options -By default the options fformat='feather' and keep_csv=True are used. +By default `dynamic_data_compiler` uses `fformat='parquet'`, `keep_csv=False`, and `keep_zip=True`. That is: -If the option fformat='csv' is used then no feather files will be created, and all caching will be done using CSVs. +- The original AEMO CSVs are extracted, converted to parquet, and then **deleted** (lean cache). Parquet has excellent compression characteristics and good compatibility with packages for handling large on-memory/cluster datasets (e.g. Dask) — this helps with local storage (especially for Causer Pays data) and file size for version control. +- The downloaded AEMO archive **zips are kept** on disk so that rebuilding the cache or switching format later does not require re-downloading from AEMO (see [#56](https://github.com/UNSW-CEEM/NEMOSIS/issues/56)). + +You can mix and match these two flags to control what is retained: + +| `keep_csv` | `keep_zip` | What stays in `raw_data_cache` | +|---|---|---| +| `False` (default) | `True` (default) | parquet/feather + zip | +| `False` | `False` | parquet/feather only — leanest cache | +| `True` | `False` | parquet/feather + CSV | +| `True` | `True` | parquet/feather + CSV + zip — full raw retention | + +For example, to keep neither the CSV nor the zip after the parquet file is written: + +```python +price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, + keep_csv=False, keep_zip=False) +``` + +To keep the raw AEMO CSVs alongside the parquet files (e.g. for an external tool that consumes the original CSV directly): + +```python +price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, keep_csv=True) +``` + +If the option `fformat='csv'` is used then no parquet files will be created, and all caching will be done using CSVs. ```python price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, fformat='csv') ``` -If you supply fformat='feather', the original AEMO CSVs will still be cached by default. To save disk space but still ensure your data will work with the API & GUI, use `keep_csv=False` in combination with `fformat='feather'` (which is the default option). This will delete the AEMO CSVs after the feather file is created. +If the option `fformat='feather'` is provided then no parquet files will be created, and a feather file will be used instead. Feather may give faster read/write than parquet, at the cost of larger files on disk. ```python -price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, keep_csv=False) +price_data = dynamic_data_compiler(start_time, end_time, table, raw_data_cache, fformat='feather') ``` -If the option `fformat='parquet'` is provided then no feather files will be created, and a parquet file will be used instead. -While feather might have faster read/write, parquet has excellent compression characteristics and good compatability with packages for handling large on-memory/cluster datasets (e.g. Dask). This helps with local storage (especially for Causer Pays data) and file size for version control. +`keep_csv` and `keep_zip` are also accepted by `cache_compiler` with the same defaults and meaning. ##### Cache compiler This may be useful if you're using NEMOSIS to build a data cache, but then process the cache using other packages or applications. It is particularly useful because `cache_compiler` will infer the data types of the columns before saving to parquet or feather, thereby eliminating the need to type convert data that is obtained using `dynamic_data_compiler`. -`cache_compiler` can be used to compile a cache of parquet or feather files. Parquet will likely be smaller, but feather can be read faster. `cache_compiler` will not run if it detects the appropriate files in the `raw_data_cache` directory. Otherwise, it will download CSVs, covert to the requested format and then delete the CSVs. It does not return any data, unlike `dynamic_data_compiler`. +`cache_compiler` can be used to compile a cache of parquet or feather files (parquet by default; pass `fformat='feather'` to switch). `cache_compiler` will not run if it detects the appropriate files in the `raw_data_cache` directory. Otherwise, it will download zip archives from AEMO, extract the CSVs, convert them to the requested format, and then delete the CSVs. The downloaded zips are kept by default (`keep_zip=True`); pass `keep_zip=False` to delete them as well. `cache_compiler` does not return any data, unlike `dynamic_data_compiler`. The example below downloads parquet data into the cache. ```python from nemosis import cache_compiler -cache_compiler(start_time, end_time, table, raw_data_cache, fformat='parquet') +cache_compiler(start_time, end_time, table, raw_data_cache) ``` ##### Accessing additional table columns @@ -199,9 +223,10 @@ cache_compiler(start_time, end_time, table, raw_data_cache, fformat='parquet') By default NEMOSIS only includes a subset of an AEMO table's columns, the full set of columns are listed in the [MMS Data Model Reports](https://visualisations.aemo.com.au/aemo/di-help/Content/Data_Model/MMS_Data_Model.htm), or can be seen by inspecting the CSVs in the raw data cache. Users of the python interface can add additional -columns as shown below. If you using a feather or parquet based cache the rebuild option should be set to -true so the additional columns are added to the cache files when they are rebuilt. This method of adding additional -columns should also work with the `cache_compiler` function. +columns as shown below. If you are using a feather or parquet based cache the rebuild option should be set to +true so the additional columns are added to the cache files when they are rebuilt. With the default `keep_zip=True`, +the rebuild re-extracts the CSV from the cached zip rather than re-downloading from AEMO. This method of adding +additional columns should also work with the `cache_compiler` function. ```python from nemosis import dynamic_data_compiler diff --git a/src/nemosis/data_fetch_methods.py b/src/nemosis/data_fetch_methods.py index 34da3ed..009caa6 100644 --- a/src/nemosis/data_fetch_methods.py +++ b/src/nemosis/data_fetch_methods.py @@ -23,7 +23,7 @@ def dynamic_data_compiler( select_columns=None, filter_cols=None, filter_values=None, - fformat="feather", + fformat="parquet", keep_csv=False, keep_zip=True, parse_data_types=True, @@ -173,7 +173,7 @@ def cache_compiler( table_name, raw_data_location, select_columns=None, - fformat="feather", + fformat="parquet", rebuild=False, keep_csv=False, keep_zip=True, @@ -596,7 +596,7 @@ def _dynamic_data_fetch_loop( raw_data_location, select_columns, date_filter, - fformat="feather", + fformat="parquet", keep_csv=False, keep_zip=True, caching_mode=False, diff --git a/src/nemosis/defaults.py b/src/nemosis/defaults.py index 186754f..10d1a64 100644 --- a/src/nemosis/defaults.py +++ b/src/nemosis/defaults.py @@ -144,7 +144,7 @@ current_data_page_urls = { "BIDDING": "Reports/Current/Bidmove_Complete/", "DAILY_REGION_SUMMARY": "/Reports/Current/Daily_Reports/", - "NEXT_DAY_DISPATCHLOAD": "/Reports/Current/NEXT_DAY_DISPATCH/", + "NEXT_DAY_DISPATCHLOAD": "/Reports/Current/Next_Day_Dispatch/", "INTERMITTENT_GEN_SCADA": "/Reports/Current/Next_Day_Intermittent_Gen_Scada/" } diff --git a/tests/end_to_end_table_tests/test_cache_compiler.py b/tests/end_to_end_table_tests/test_cache_compiler.py index 0d5e0a2..578dd78 100644 --- a/tests/end_to_end_table_tests/test_cache_compiler.py +++ b/tests/end_to_end_table_tests/test_cache_compiler.py @@ -75,7 +75,7 @@ def test_creates_cache_directory_when_missing(nemosis_fixture): ) assert target.is_dir() - assert list(target.glob("*.feather")), "cache should be populated" + assert list(target.glob("*.parquet")), "cache should be populated" def test_raises_when_cache_path_is_a_file(nemosis_fixture): @@ -93,9 +93,9 @@ def test_raises_when_cache_path_is_a_file(nemosis_fixture): def test_keep_csv_true_retains_csv(nemosis_fixture): - """When cache_compiler actually has to fetch (no existing feather, so + """When cache_compiler actually has to fetch (no existing parquet, so the code path that downloads + extracts a CSV runs), `keep_csv=True` - must leave the extracted CSV on disk alongside the feather. + must leave the extracted CSV on disk alongside the parquet. rebuild=True forces the fetch path so this test isn't dependent on tmp_path being empty at start. @@ -115,7 +115,7 @@ def test_keep_csv_true_retains_csv(nemosis_fixture): def test_keep_csv_default_is_false(nemosis_fixture): """Default behaviour — keep_csv=False removes the extracted CSV - after the typed feather is written, leaving only the feather in + after the typed parquet is written, leaving only the parquet in the cache. Users who want raw retention opt in via keep_csv=True.""" cache_compiler( start_time=START, end_time=END, @@ -128,18 +128,18 @@ def test_keep_csv_default_is_false(nemosis_fixture): assert not csv_files, f"default keep_csv=False should remove CSV; found: {csv_files}" -def test_existing_feather_means_no_csv_is_fetched(nemosis_fixture): - """If the feather is already in the cache, cache_compiler must take +def test_existing_parquet_means_no_csv_is_fetched(nemosis_fixture): + """If the parquet is already in the cache, cache_compiler must take the "already compiled" short-circuit and not fetch a CSV — keep_csv is only about retaining a CSV we actually downloaded, not about - creating one out of thin air. Pre-populate empty feather files at + creating one out of thin air. Pre-populate empty parquet files at the expected filenames (in caching_mode the existence check skips the read), call cache_compiler without rebuild, and verify no CSV appeared.""" # April + May because NEMOSIS uses a 1-day buffer-back, so a query # starting 2018-05-01 also scans the 2018-04 archive. for month in ("201804", "201805"): - (nemosis_fixture / f"PUBLIC_DVD_DISPATCHPRICE_{month}010000.feather").touch() + (nemosis_fixture / f"PUBLIC_DVD_DISPATCHPRICE_{month}010000.parquet").touch() cache_compiler( start_time=START, end_time=END, @@ -151,7 +151,7 @@ def test_existing_feather_means_no_csv_is_fetched(nemosis_fixture): csv_files = list(nemosis_fixture.glob("*DISPATCHPRICE*.[Cc][Ss][Vv]")) assert not csv_files, ( "keep_csv=True should NOT cause a CSV to be created when the " - "feather already exists — the CSV branch must not run at all" + "parquet already exists — the CSV branch must not run at all" ) @@ -204,7 +204,7 @@ def test_keep_zip_false_removes_zip(nemosis_fixture): def test_cached_zip_extracts_without_network(nemosis_fixture, monkeypatch): """The #56 benefit in action: with keep_zip=True on a first call, - a subsequent call that needs the same CSV but finds the feather + a subsequent call that needs the same CSV but finds the parquet missing should re-extract from the cached zip locally without hitting nemweb. Proves it by breaking the AEMO URL after the first call — if the second call tries to fetch, it would 404 against the @@ -217,14 +217,14 @@ def test_cached_zip_extracts_without_network(nemosis_fixture, monkeypatch): rebuild=True, keep_zip=True, ) - feather_files = list(nemosis_fixture.glob("*.feather")) + parquet_files = list(nemosis_fixture.glob("*.parquet")) zip_files = list(nemosis_fixture.glob("*.zip")) - assert feather_files and zip_files + assert parquet_files and zip_files - # Delete the feather so the loop has to call _download_data again, + # Delete the parquet so the loop has to call _download_data again, # but leave the zip in place so the lower-level download_to_path # short-circuits on the cached file. - for f in feather_files: + for f in parquet_files: f.unlink() # Point the URL at a dead address — any actual network call would fail. @@ -239,8 +239,8 @@ def test_cached_zip_extracts_without_network(nemosis_fixture, monkeypatch): keep_zip=True, ) - # Feather rebuilt from the cached zip - assert list(nemosis_fixture.glob("*.feather")), "cache should be rebuilt from cached zip" + # Parquet rebuilt from the cached zip + assert list(nemosis_fixture.glob("*.parquet")), "cache should be rebuilt from cached zip" @pytest.mark.parametrize("fformat", ["feather", "parquet"]) diff --git a/tests/end_to_end_table_tests/test_daily_region_summary.py b/tests/end_to_end_table_tests/test_daily_region_summary.py index d23fa54..8dc9105 100644 --- a/tests/end_to_end_table_tests/test_daily_region_summary.py +++ b/tests/end_to_end_table_tests/test_daily_region_summary.py @@ -17,8 +17,8 @@ def test_recent_day_returns_rows_for_filtered_regions(nemosis_fixture): data = dynamic_data_compiler( - start_time="2026/03/15 00:00:00", - end_time="2026/03/15 01:00:00", + start_time="2026/05/15 00:00:00", + end_time="2026/05/15 01:00:00", table_name="DAILY_REGION_SUMMARY", raw_data_location=str(nemosis_fixture), select_columns=["SETTLEMENTDATE", "REGIONID", "TOTALDEMAND"], @@ -38,31 +38,31 @@ def test_recent_day_returns_rows_for_filtered_regions(nemosis_fixture): def test_end_market_day_returns_0400_row_from_previous_file(nemosis_fixture): """[03:55, 04:00]: filter is start-exclusive end-inclusive, so only - the 04:00 row qualifies — and it lives in the `20260314` daily file, - not `20260315`. Non-empty proves buffer-back fired correctly.""" + the 04:00 row qualifies — and it lives in the `20260514` daily file, + not `20260515`. Non-empty proves buffer-back fired correctly.""" data = dynamic_data_compiler( - start_time="2026/03/15 03:55:00", - end_time="2026/03/15 04:00:00", + start_time="2026/05/15 03:55:00", + end_time="2026/05/15 04:00:00", table_name="DAILY_REGION_SUMMARY", raw_data_location=str(nemosis_fixture), select_columns=["SETTLEMENTDATE", "REGIONID", "TOTALDEMAND"], ) - assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-03-15 04:00:00")} + assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-05-15 04:00:00")} assert set(data["REGIONID"]) == {"SA1", "NSW1"} assert not data.duplicated(["SETTLEMENTDATE", "REGIONID"]).any() def test_start_market_day_returns_0405_row_from_current_file(nemosis_fixture): """[04:00, 04:05]: start-exclusive means 04:00 is dropped; 04:05 is - the first row of the `20260315` daily file. Tests the stitch from + the first row of the `20260515` daily file. Tests the stitch from the consumer side — the current day's file must be fetched.""" data = dynamic_data_compiler( - start_time="2026/03/15 04:00:00", - end_time="2026/03/15 04:05:00", + start_time="2026/05/15 04:00:00", + end_time="2026/05/15 04:05:00", table_name="DAILY_REGION_SUMMARY", raw_data_location=str(nemosis_fixture), select_columns=["SETTLEMENTDATE", "REGIONID", "TOTALDEMAND"], ) - assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-03-15 04:05:00")} + assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-05-15 04:05:00")} assert set(data["REGIONID"]) == {"SA1", "NSW1"} assert not data.duplicated(["SETTLEMENTDATE", "REGIONID"]).any() diff --git a/tests/end_to_end_table_tests/test_datetime_inputs.py b/tests/end_to_end_table_tests/test_datetime_inputs.py index 050ffa5..87e5c91 100644 --- a/tests/end_to_end_table_tests/test_datetime_inputs.py +++ b/tests/end_to_end_table_tests/test_datetime_inputs.py @@ -72,13 +72,13 @@ def test_dynamic_data_compiler_accepts_date(nemosis_fixture): def test_cache_compiler_accepts_datetime(nemosis_fixture): """cache_compiler must accept the same input shapes as dynamic_data_compiler — just confirms no exception and that a - feather lands in the cache.""" + parquet lands in the cache.""" cache_compiler( start_time=START_DT, end_time=END_DT, table_name="DISPATCHPRICE", raw_data_location=str(nemosis_fixture), ) - assert list(nemosis_fixture.glob("*DISPATCHPRICE*.feather")), ( + assert list(nemosis_fixture.glob("*DISPATCHPRICE*.parquet")), ( "cache should be populated when datetime inputs are supplied" ) @@ -90,7 +90,7 @@ def test_cache_compiler_accepts_date(nemosis_fixture): table_name="DISPATCHPRICE", raw_data_location=str(nemosis_fixture), ) - assert list(nemosis_fixture.glob("*DISPATCHPRICE*.feather")) + assert list(nemosis_fixture.glob("*DISPATCHPRICE*.parquet")) # --------------------------------------------------------------------------- diff --git a/tests/end_to_end_table_tests/test_intermittent_gen_scada.py b/tests/end_to_end_table_tests/test_intermittent_gen_scada.py index 91179ac..f61d983 100644 --- a/tests/end_to_end_table_tests/test_intermittent_gen_scada.py +++ b/tests/end_to_end_table_tests/test_intermittent_gen_scada.py @@ -13,8 +13,8 @@ def test_recent_day_returns_rows_for_fixtured_duids(nemosis_fixture): data = dynamic_data_compiler( - start_time="2026/03/15 00:00:00", - end_time="2026/03/15 01:00:00", + start_time="2026/05/15 00:00:00", + end_time="2026/05/15 01:00:00", table_name="INTERMITTENT_GEN_SCADA", raw_data_location=str(nemosis_fixture), select_columns=["RUN_DATETIME", "DUID", "SCADA_VALUE"], @@ -32,13 +32,13 @@ def test_recent_day_returns_rows_for_fixtured_duids(nemosis_fixture): def test_end_market_day_returns_0400_row_from_previous_file(nemosis_fixture): data = dynamic_data_compiler( - start_time="2026/03/15 03:55:00", - end_time="2026/03/15 04:00:00", + start_time="2026/05/15 03:55:00", + end_time="2026/05/15 04:00:00", table_name="INTERMITTENT_GEN_SCADA", raw_data_location=str(nemosis_fixture), select_columns=["RUN_DATETIME", "DUID", "SCADA_TYPE", "SCADA_VALUE"], ) - assert set(data["RUN_DATETIME"].unique()) == {pd.Timestamp("2026-03-15 04:00:00")} + assert set(data["RUN_DATETIME"].unique()) == {pd.Timestamp("2026-05-15 04:00:00")} assert set(data["DUID"]) == {"HDWF2"} assert set(data["SCADA_TYPE"]) == {"ELAV", "LOCL"} assert not data.duplicated(["RUN_DATETIME", "DUID", "SCADA_TYPE"]).any() @@ -46,13 +46,13 @@ def test_end_market_day_returns_0400_row_from_previous_file(nemosis_fixture): def test_start_market_day_returns_0405_row_from_current_file(nemosis_fixture): data = dynamic_data_compiler( - start_time="2026/03/15 04:00:00", - end_time="2026/03/15 04:05:00", + start_time="2026/05/15 04:00:00", + end_time="2026/05/15 04:05:00", table_name="INTERMITTENT_GEN_SCADA", raw_data_location=str(nemosis_fixture), select_columns=["RUN_DATETIME", "DUID", "SCADA_TYPE", "SCADA_VALUE"], ) - assert set(data["RUN_DATETIME"].unique()) == {pd.Timestamp("2026-03-15 04:05:00")} + assert set(data["RUN_DATETIME"].unique()) == {pd.Timestamp("2026-05-15 04:05:00")} assert set(data["DUID"]) == {"HDWF2"} assert set(data["SCADA_TYPE"]) == {"ELAV", "LOCL"} assert not data.duplicated(["RUN_DATETIME", "DUID", "SCADA_TYPE"]).any() diff --git a/tests/end_to_end_table_tests/test_next_day_dispatch_load.py b/tests/end_to_end_table_tests/test_next_day_dispatch_load.py index a52b1e1..0c43259 100644 --- a/tests/end_to_end_table_tests/test_next_day_dispatch_load.py +++ b/tests/end_to_end_table_tests/test_next_day_dispatch_load.py @@ -13,8 +13,8 @@ def test_recent_day_returns_rows_for_fixtured_duids(nemosis_fixture): data = dynamic_data_compiler( - start_time="2026/03/15 00:00:00", - end_time="2026/03/15 01:00:00", + start_time="2026/05/15 00:00:00", + end_time="2026/05/15 01:00:00", table_name="NEXT_DAY_DISPATCHLOAD", raw_data_location=str(nemosis_fixture), select_columns=["SETTLEMENTDATE", "DUID", "INITIALMW"], @@ -30,25 +30,25 @@ def test_recent_day_returns_rows_for_fixtured_duids(nemosis_fixture): def test_end_market_day_returns_0400_row_from_previous_file(nemosis_fixture): data = dynamic_data_compiler( - start_time="2026/03/15 03:55:00", - end_time="2026/03/15 04:00:00", + start_time="2026/05/15 03:55:00", + end_time="2026/05/15 04:00:00", table_name="NEXT_DAY_DISPATCHLOAD", raw_data_location=str(nemosis_fixture), select_columns=["SETTLEMENTDATE", "DUID", "INITIALMW"], ) - assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-03-15 04:00:00")} + assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-05-15 04:00:00")} assert set(data["DUID"]) <= {"AGLHAL", "HDWF2"} assert not data.duplicated(["SETTLEMENTDATE", "DUID"]).any() def test_start_market_day_returns_0405_row_from_current_file(nemosis_fixture): data = dynamic_data_compiler( - start_time="2026/03/15 04:00:00", - end_time="2026/03/15 04:05:00", + start_time="2026/05/15 04:00:00", + end_time="2026/05/15 04:05:00", table_name="NEXT_DAY_DISPATCHLOAD", raw_data_location=str(nemosis_fixture), select_columns=["SETTLEMENTDATE", "DUID", "INITIALMW"], ) - assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-03-15 04:05:00")} + assert set(data["SETTLEMENTDATE"].unique()) == {pd.Timestamp("2026-05-15 04:05:00")} assert set(data["DUID"]) <= {"AGLHAL", "HDWF2"} assert not data.duplicated(["SETTLEMENTDATE", "DUID"]).any() diff --git a/tests/fixtures/build.py b/tests/fixtures/build.py index 84c6ae3..9ee3624 100644 --- a/tests/fixtures/build.py +++ b/tests/fixtures/build.py @@ -289,7 +289,7 @@ def make_index_html(filenames: list[str], url_prefix: str) -> str: "stub_template": "PUBLIC_DAILY_{ymd}", "sections_to_keep": {1: {"REGIONID": spec.REGIONS}}, }, - "Reports/Current/NEXT_DAY_DISPATCH/": { + "Reports/Current/Next_Day_Dispatch/": { "stub_template": "PUBLIC_NEXT_DAY_DISPATCH_{ymd}", "sections_to_keep": {0: {"DUID": spec.DUIDS}}, }, @@ -302,7 +302,7 @@ def make_index_html(filenames: list[str], url_prefix: str) -> str: # Which NEMOSIS table maps to which scrape file. TABLE_SCRAPE_FILE = { "DAILY_REGION_SUMMARY": "Reports/Current/Daily_Reports/", - "NEXT_DAY_DISPATCHLOAD": "Reports/Current/NEXT_DAY_DISPATCH/", + "NEXT_DAY_DISPATCHLOAD": "Reports/Current/Next_Day_Dispatch/", "INTERMITTENT_GEN_SCADA": "Reports/Current/Next_Day_Intermittent_Gen_Scada/", } diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDDAYOFFER_D%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDDAYOFFER_D%23FILE01%23202603010000.zip new file mode 100644 index 0000000..ec5a4f4 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDDAYOFFER_D%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDPEROFFER_D%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDPEROFFER_D%23FILE01%23202603010000.zip new file mode 100644 index 0000000..786cab5 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDPEROFFER_D%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHCONSTRAINT%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHCONSTRAINT%23FILE01%23202603010000.zip new file mode 100644 index 0000000..469a4d5 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHCONSTRAINT%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHINTERCONNECTORRES%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHINTERCONNECTORRES%23FILE01%23202603010000.zip new file mode 100644 index 0000000..ea00993 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHINTERCONNECTORRES%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHLOAD%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHLOAD%23FILE01%23202603010000.zip new file mode 100644 index 0000000..148404f Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHLOAD%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23202603010000.zip new file mode 100644 index 0000000..e62bcf7 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHREGIONSUM%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHREGIONSUM%23FILE01%23202603010000.zip new file mode 100644 index 0000000..79b000a Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHREGIONSUM%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCH_UNIT_SCADA%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCH_UNIT_SCADA%23FILE01%23202603010000.zip new file mode 100644 index 0000000..b71557e Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCH_UNIT_SCADA%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23MNSP_DAYOFFER%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23MNSP_DAYOFFER%23FILE01%23202603010000.zip new file mode 100644 index 0000000..969fdba Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23MNSP_DAYOFFER%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23ROOFTOP_PV_ACTUAL%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23ROOFTOP_PV_ACTUAL%23FILE01%23202603010000.zip new file mode 100644 index 0000000..14ca3d6 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23ROOFTOP_PV_ACTUAL%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGINTERCONNECT%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGINTERCONNECT%23FILE01%23202603010000.zip new file mode 100644 index 0000000..c442935 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGINTERCONNECT%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGPRICE%23FILE01%23202603010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGPRICE%23FILE01%23202603010000.zip new file mode 100644 index 0000000..155ce37 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_03/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGPRICE%23FILE01%23202603010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDDAYOFFER_D%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDDAYOFFER_D%23FILE01%23202604010000.zip new file mode 100644 index 0000000..b09dfd8 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDDAYOFFER_D%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDPEROFFER_D%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDPEROFFER_D%23FILE01%23202604010000.zip new file mode 100644 index 0000000..82497bc Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23BIDPEROFFER_D%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHCONSTRAINT%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHCONSTRAINT%23FILE01%23202604010000.zip new file mode 100644 index 0000000..9a745d6 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHCONSTRAINT%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHINTERCONNECTORRES%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHINTERCONNECTORRES%23FILE01%23202604010000.zip new file mode 100644 index 0000000..cc7932a Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHINTERCONNECTORRES%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHLOAD%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHLOAD%23FILE01%23202604010000.zip new file mode 100644 index 0000000..376874a Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHLOAD%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23202604010000.zip new file mode 100644 index 0000000..6b86dc2 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHPRICE%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHREGIONSUM%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHREGIONSUM%23FILE01%23202604010000.zip new file mode 100644 index 0000000..87a75a4 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCHREGIONSUM%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCH_UNIT_SCADA%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCH_UNIT_SCADA%23FILE01%23202604010000.zip new file mode 100644 index 0000000..c964354 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23DISPATCH_UNIT_SCADA%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23MNSP_DAYOFFER%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23MNSP_DAYOFFER%23FILE01%23202604010000.zip new file mode 100644 index 0000000..e5e6229 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23MNSP_DAYOFFER%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23ROOFTOP_PV_ACTUAL%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23ROOFTOP_PV_ACTUAL%23FILE01%23202604010000.zip new file mode 100644 index 0000000..2a0b6c8 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23ROOFTOP_PV_ACTUAL%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGINTERCONNECT%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGINTERCONNECT%23FILE01%23202604010000.zip new file mode 100644 index 0000000..6a366b4 Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGINTERCONNECT%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGPRICE%23FILE01%23202604010000.zip b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGPRICE%23FILE01%23202604010000.zip new file mode 100644 index 0000000..992a59a Binary files /dev/null and b/tests/fixtures/data/Data_Archive/Wholesale_Electricity/MMSDM/2026/MMSDM_2026_04/MMSDM_Historical_Data_SQLLoader/DATA/PUBLIC_ARCHIVE%23TRADINGPRICE%23FILE01%23202604010000.zip differ diff --git a/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202603140000_20260315040503.zip b/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202603140000_20260315040503.zip deleted file mode 100644 index 4ef62be..0000000 Binary files a/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202603140000_20260315040503.zip and /dev/null differ diff --git a/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202603150000_20260316040504.zip b/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202603150000_20260316040504.zip deleted file mode 100644 index 4522146..0000000 Binary files a/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202603150000_20260316040504.zip and /dev/null differ diff --git a/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202605140000_20260515040504.zip b/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202605140000_20260515040504.zip new file mode 100644 index 0000000..86414fb Binary files /dev/null and b/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202605140000_20260515040504.zip differ diff --git a/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202605150000_20260516040505.zip b/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202605150000_20260516040505.zip new file mode 100644 index 0000000..76c94d2 Binary files /dev/null and b/tests/fixtures/data/Reports/Current/Daily_Reports/PUBLIC_DAILY_202605150000_20260516040505.zip differ diff --git a/tests/fixtures/data/Reports/Current/Daily_Reports/index.html b/tests/fixtures/data/Reports/Current/Daily_Reports/index.html index 73c53b3..f552f57 100644 --- a/tests/fixtures/data/Reports/Current/Daily_Reports/index.html +++ b/tests/fixtures/data/Reports/Current/Daily_Reports/index.html @@ -1,4 +1,4 @@
-PUBLIC_DAILY_202603140000_20260315040503.zip -PUBLIC_DAILY_202603150000_20260316040504.zip +PUBLIC_DAILY_202605140000_20260515040504.zip +PUBLIC_DAILY_202605150000_20260516040505.zip \ No newline at end of file diff --git a/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/PUBLIC_NEXT_DAY_DISPATCH_20260314_0000000507927925.zip b/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/PUBLIC_NEXT_DAY_DISPATCH_20260314_0000000507927925.zip deleted file mode 100644 index c302d57..0000000 Binary files a/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/PUBLIC_NEXT_DAY_DISPATCH_20260314_0000000507927925.zip and /dev/null differ diff --git a/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/PUBLIC_NEXT_DAY_DISPATCH_20260315_0000000508079749.zip b/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/PUBLIC_NEXT_DAY_DISPATCH_20260315_0000000508079749.zip deleted file mode 100644 index dec3fef..0000000 Binary files a/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/PUBLIC_NEXT_DAY_DISPATCH_20260315_0000000508079749.zip and /dev/null differ diff --git a/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/index.html b/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/index.html deleted file mode 100644 index cd113f4..0000000 --- a/tests/fixtures/data/Reports/Current/NEXT_DAY_DISPATCH/index.html +++ /dev/null @@ -1,4 +0,0 @@ - -PUBLIC_NEXT_DAY_DISPATCH_20260314_0000000507927925.zip -PUBLIC_NEXT_DAY_DISPATCH_20260315_0000000508079749.zip - \ No newline at end of file diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/PUBLIC_NEXT_DAY_DISPATCH_20260514_0000000517721207.zip b/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/PUBLIC_NEXT_DAY_DISPATCH_20260514_0000000517721207.zip new file mode 100644 index 0000000..4b8c2c7 Binary files /dev/null and b/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/PUBLIC_NEXT_DAY_DISPATCH_20260514_0000000517721207.zip differ diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/PUBLIC_NEXT_DAY_DISPATCH_20260515_0000000517880947.zip b/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/PUBLIC_NEXT_DAY_DISPATCH_20260515_0000000517880947.zip new file mode 100644 index 0000000..2f50379 Binary files /dev/null and b/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/PUBLIC_NEXT_DAY_DISPATCH_20260515_0000000517880947.zip differ diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/index.html b/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/index.html new file mode 100644 index 0000000..53d74e7 --- /dev/null +++ b/tests/fixtures/data/Reports/Current/Next_Day_Dispatch/index.html @@ -0,0 +1,4 @@ + +PUBLIC_NEXT_DAY_DISPATCH_20260514_0000000517721207.zip +PUBLIC_NEXT_DAY_DISPATCH_20260515_0000000517880947.zip + \ No newline at end of file diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260314_0000000507927930.zip b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260314_0000000507927930.zip deleted file mode 100644 index f8c6b80..0000000 Binary files a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260314_0000000507927930.zip and /dev/null differ diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260315_0000000508079754.zip b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260315_0000000508079754.zip deleted file mode 100644 index c902895..0000000 Binary files a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260315_0000000508079754.zip and /dev/null differ diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260514_0000000517721212.zip b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260514_0000000517721212.zip new file mode 100644 index 0000000..c083fd6 Binary files /dev/null and b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260514_0000000517721212.zip differ diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260515_0000000517880952.zip b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260515_0000000517880952.zip new file mode 100644 index 0000000..6f6ae63 Binary files /dev/null and b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260515_0000000517880952.zip differ diff --git a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/index.html b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/index.html index 9c56b6d..b999c8f 100644 --- a/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/index.html +++ b/tests/fixtures/data/Reports/Current/Next_Day_Intermittent_Gen_Scada/index.html @@ -1,4 +1,4 @@ -PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260314_0000000507927930.zip -PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260315_0000000508079754.zip +PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260514_0000000517721212.zip +PUBLIC_NEXT_DAY_INTERMITTENT_GEN_SCADA_20260515_0000000517880952.zip \ No newline at end of file diff --git a/tests/fixtures/spec.py b/tests/fixtures/spec.py index aecd4eb..09a103a 100644 --- a/tests/fixtures/spec.py +++ b/tests/fixtures/spec.py @@ -27,7 +27,8 @@ "2024-08": date(2024, 8, 1), # PUBLIC_DVD_ → PUBLIC_ARCHIVE# cutover; `into` straddles the format change "2024-09": date(2024, 9, 1), # first safely-past-cutover month; used by bidding tables (see note) "2025-01": date(2025, 1, 1), # year boundary, PUBLIC_ARCHIVE# format, post-bidding-gap - "recent": date(2026, 3, 15), # inside AEMO's current-data scrape window (see note) + "2026-04": date(2026, 4, 1), # general recent-month coverage in PUBLIC_ARCHIVE# format + "recent": date(2026, 5, 15), # inside AEMO's current-data scrape window (see note) } # Year-boundary eras (2020-01, 2022-01, 2025-01) exist for the boundary test # matrix in tests/end_to_end_table_tests/_boundaries.py — they exercise @@ -65,12 +66,12 @@ # More tables get added as each one lands end-to-end. DYNAMIC_TABLES = { # Five-minute dispatch tables — the workhorse set. - "DISPATCHPRICE": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01"], "filter": {"REGIONID": REGIONS}}, - "DISPATCHLOAD": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01"], "filter": {"DUID": DUIDS}}, - "DISPATCH_UNIT_SCADA": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01"], "filter": {"DUID": DUIDS}}, - "DISPATCHREGIONSUM": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01"], "filter": {"REGIONID": REGIONS}}, - "DISPATCHINTERCONNECTORRES": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01"], "filter": {"INTERCONNECTORID": INTERCONNECTORS}}, - "DISPATCHCONSTRAINT": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01"], "filter": {"CONSTRAINTID": CONSTRAINTS}}, + "DISPATCHPRICE": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01", "2026-04"], "filter": {"REGIONID": REGIONS}}, + "DISPATCHLOAD": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01", "2026-04"], "filter": {"DUID": DUIDS}}, + "DISPATCH_UNIT_SCADA": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01", "2026-04"], "filter": {"DUID": DUIDS}}, + "DISPATCHREGIONSUM": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01", "2026-04"], "filter": {"REGIONID": REGIONS}}, + "DISPATCHINTERCONNECTORRES": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01", "2026-04"], "filter": {"INTERCONNECTORID": INTERCONNECTORS}}, + "DISPATCHCONSTRAINT": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2024-08", "2025-01", "2026-04"], "filter": {"CONSTRAINTID": CONSTRAINTS}}, # Trading tables. TRADINGPRICE/TRADINGINTERCONNECT switched from 30-min to # 5-min at 2021-10-01 (the 5MS reform cutover); TRADINGLOAD/TRADINGREGIONSUM @@ -80,8 +81,8 @@ # dedicated stride-transition test in test_trading_price.py. It is not # listed in `eras`, so the boundary tests skip it — the 5MS cutover # month is covered by that dedicated test instead. - "TRADINGPRICE": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2022-06", "2024-08", "2025-01"], "extra_eras": ["2021-10"], "filter": {"REGIONID": REGIONS}}, - "TRADINGINTERCONNECT": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2022-06", "2024-08", "2025-01"], "extra_eras": ["2021-10"], "filter": {"INTERCONNECTORID": INTERCONNECTORS}}, + "TRADINGPRICE": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2022-06", "2024-08", "2025-01", "2026-04"], "extra_eras": ["2021-10"], "filter": {"REGIONID": REGIONS}}, + "TRADINGINTERCONNECT": {"eras": ["2018-05", "2020-01", "2021-05", "2022-01", "2022-06", "2024-08", "2025-01", "2026-04"], "extra_eras": ["2021-10"], "filter": {"INTERCONNECTORID": INTERCONNECTORS}}, # TRADINGLOAD/TRADINGREGIONSUM were discontinued before 2022 — 2022-01 returns 404. "TRADINGLOAD": {"eras": ["2018-05", "2020-01", "2021-05"], "filter": {"DUID": DUIDS}}, "TRADINGREGIONSUM": {"eras": ["2018-05", "2020-01", "2021-05"], "filter": {"REGIONID": REGIONS}}, @@ -94,8 +95,8 @@ # filename cutover. Bidding tables stay on 2024-09: 2024-07's bid archive # sits inside the publishing gap and a 2024-08-era prev-month buffer would # 404 at test time. - "BIDPEROFFER_D": {"eras": ["2018-05", "2020-01", "2021-02", "2024-09", "2025-01"], "filter": {"DUID": DUIDS}}, - "BIDDAYOFFER_D": {"eras": ["2018-05", "2020-01", "2021-02", "2024-09", "2025-01"], "filter": {"DUID": DUIDS}}, + "BIDPEROFFER_D": {"eras": ["2018-05", "2020-01", "2021-02", "2024-09", "2025-01", "2026-04"], "filter": {"DUID": DUIDS}}, + "BIDDAYOFFER_D": {"eras": ["2018-05", "2020-01", "2021-02", "2024-09", "2025-01", "2026-04"], "filter": {"DUID": DUIDS}}, # MNSP bidding — same shape as BIDPEROFFER_D / BIDDAYOFFER_D but keyed on # LINKID (MNSP interconnectors bid as directional links). @@ -114,7 +115,7 @@ # standard bidding era set applies — same bidding-gap shape as BIDPEROFFER_D # / BIDDAYOFFER_D above (no 2022-01, recovery at 2024-09). "MNSP_PEROFFER": {"eras": ["2018-05", "2020-01", "2021-02"], "filter": {"LINKID": MNSP_LINKS}}, - "MNSP_DAYOFFER": {"eras": ["2018-05", "2020-01", "2021-05", "2024-09", "2025-01"], "filter": {"LINKID": MNSP_LINKS}}, + "MNSP_DAYOFFER": {"eras": ["2018-05", "2020-01", "2021-05", "2024-09", "2025-01", "2026-04"], "filter": {"LINKID": MNSP_LINKS}}, # Scrape-only tables — all live on AEMO's rolling current-data pages. "DAILY_REGION_SUMMARY": {"eras": ["recent"], "filter": {"REGIONID": REGIONS}}, @@ -122,7 +123,7 @@ "INTERMITTENT_GEN_SCADA": {"eras": ["recent"], "filter": {"DUID": DUIDS}}, # Rooftop PV — introduced mid-2019. - "ROOFTOP_PV_ACTUAL": {"eras": ["2020-01", "2021-05", "2022-01", "2024-08", "2025-01"], "filter": {"REGIONID": REGIONS}}, + "ROOFTOP_PV_ACTUAL": {"eras": ["2020-01", "2021-05", "2022-01", "2024-08", "2025-01", "2026-04"], "filter": {"REGIONID": REGIONS}}, # Effective-date config tables. These publish sparsely — records are # configuration changes, not time-series observations — and NEMOSIS's