diff --git a/scripts/R_programmer_guide.md b/scripts/R_programmer_guide.md new file mode 100644 index 0000000..3c42fdc --- /dev/null +++ b/scripts/R_programmer_guide.md @@ -0,0 +1,135 @@ +# R Programmer Guide: Using the Python Resolver and Manifest Builder + +This guide shows how R users can call the shared Python utilities (`resolve_path` and +`build_manifest`) from R via the `reticulate` package. It includes examples, common +patterns, and links to the bundled test harnesses you can run to verify return shapes +before wiring them into your pipelines. + +Location of test harnesses +- `scripts/test_harness/test_resolve_path.py` — dry-run resolver outputs (indicator/shared) +- `scripts/test_harness/test_build_manifest.py` — dry-run manifest builder outputs + +Note: Python dictionaries returned by these modules map to named lists in R when +imported via `reticulate`. Access elements with the `$` operator. + +--- + +## 1. Using `resolve_path` from R + +Purpose: obtain `root` and `relative` path components for indicator downloads or shared +assets so your R ingestion code can be environment-agnostic. + +Example (reticulate): + +```R +library(reticulate) +resolve_path <- import_from_path("resolve_path", path = "./scripts/shared") + +# Simple download lookup (indicator) +parts <- resolve_path$get_download_path("o3", "1.0", "raw_o3", "local") +print(parts$root) # e.g. "./pipeline/o3/" +print(parts$relative) # e.g. "v1.0/downloads/..." + +# Shared asset lookup (preprocessed template). No interpolation is performed by resolver. +shared_ver <- resolve_path$get_dependency_version("o3", "1.0", "census_block_weights") +shared_parts <- resolve_path$get_shared_asset_path("census_block_weights", shared_ver, "preprocessed_input", environment = "local") +print(shared_parts$relative) # contains token like "...{postal}..." + +``` + +Token replacement and building a usable path: + +```R +state_postal <- "WY" +relative_filled <- gsub("\\{postal\\}", state_postal, shared_parts$relative) +full_path <- paste0(shared_parts$root, relative_filled) + +# Read locally +df <- readr::read_csv(full_path) + +# Or, for remote S3 URIs, stream using an S3-aware reader (do not use `file.path`) +``` + +Quick CLI checks (shell) to confirm what `reticulate` will receive: + +```bash +python scripts/test_harness/test_resolve_path.py --storage-mode local --type indicator --name o3 --version 1.0 --key raw_o3 +python scripts/test_harness/test_resolve_path.py --storage-mode local --type shared --name census_block_weights --version 1.0 --category preprocessed_input +``` + +These produce a small JSON object with `root` and `relative` fields; the harness adds a +`status` field for clarity. + +--- + +## 2. Using `build_manifest` from R + +Purpose: retrieve a compiled stage manifest (inputs/outputs) that already includes +`root` and `relative` entries for each declared input and output. This is useful when +you want a single manifest-driven object to drive multiple parts of a workflow. + +Example (reticulate): + +```R +library(reticulate) +build_manifest <- import_from_path("build_manifest", path = "./scripts/shared") + +manifest <- build_manifest$get_stage_manifest( + target_type = "indicator", + name = "o3", + stage = "preprocess", + version = "1.0", + environment = "local" +) + +# Inspect the returned named list +print(names(manifest)) # typically: "inputs" "outputs" +print(manifest$inputs$primary_o3$root) +print(manifest$outputs$main_tract_averages$relative) + +``` + +Quick CLI check (shell) to confirm manifest shape and contents: + +```bash +python scripts/test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0 +``` + +The harness prints a JSON object prefixed with `status` that mirrors the manifest +structure you will get via `reticulate`. + +Notes and recommendations +- The manifest entries for inputs/outputs already include `root` + `relative` so you + can construct `full_path <- paste0(entry$root, entry$relative)` in R. +- If an entry's `relative` contains tokens (e.g. `{postal}`), perform R-side + substitution with `gsub()` before concatenation. +- Avoid using `file.path()` for S3 URIs — prefer string concatenation to preserve + forward slashes. + +--- + +## 3. Where to run tests and what to expect + +- Resolver test harness: `scripts/test_harness/test_resolve_path.py` + - Use to validate `get_download_path` and `get_shared_asset_path` return shapes + - Example: `python scripts/test_harness/test_resolve_path.py --storage-mode local --type indicator --name o3 --version 1.0 --key raw_o3` + +- Manifest test harness: `scripts/test_harness/test_build_manifest.py` + - Use to validate `get_stage_manifest` returns `inputs`/`outputs` with compiled entries + - Example: `python scripts/test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0` + +I added a short R script in `scripts/test_harness/` that calls these functions via +`reticulate` and prints the R named-list shapes. Run it with `Rscript` to see what +your `reticulate` calls should return: + +```bash +Rscript scripts/test_harness/test_resolve_path_reticulate.R +``` + +The script runs a small set of resolver and manifest checks and prints JSON-formatted +outputs to stdout. It is intended as a low-risk local verification step before you +embed `reticulate` calls in production R code. + +--- + +Last updated: 2026-06-13 diff --git a/scripts/agToDo.md b/scripts/agToDo.md new file mode 100644 index 0000000..54ce781 --- /dev/null +++ b/scripts/agToDo.md @@ -0,0 +1,36 @@ +# Checklist: Configuration & Path Resolution Refactor + +## Phase 1: Environment & Schema Setup +- [x] Create a new Git branch for the refactor: `dataVersioning` +- [x] Update `scripts/o3/o3_config.json` to match the new schema structure (including `local_root_path`, `remote_root_path`, and stage definitions for inputs/outputs) +- [x] Update `scripts/shared/shared_config.json` to match the new global asset schema + +## Phase 2: Core Path Resolution (The Machinist) +- [x] Prompt AI to generate `scripts/shared/resolve_path.py` based on Section 1 specifications +- [x] Prompt AI to generate the CLI test harness `test_resolve_path.py` based on Section 2 specifications +- [x] Test `resolve_path.py` using the harness against `o3_config.json`: + - [x] Verify local resolution returns accurate `{"root": ..., "relative": ...}` dictionary + - [x] Verify remote resolution handles S3 paths correctly without Windows backslash errors +- [x] Test `resolve_path.py` using the harness against `shared_config.json` (e.g., checking dependency version resolution logic) + +## Phase 3: Workflow Mapping (The Architect) +- [x] Prompt AI to generate `scripts/shared/build_manifest.py` based on the manifest specification +- [x] Prompt AI to generate the CLI test harness `test_manifest.py` +- [x] Test `build_manifest.py` using the harness for the `o3` `preprocess` stage: + - [x] Verify that it auto-resolves global shared dependencies (e.g., pulling correct version of `census_block_weights`) + - [x] Verify that auxiliary output folders are cleanly mapped in the final dictionary + +## Phase 4: Tracer Bullet Integration (O3 Testing) +- [x] Refactor the `fetch_raw` script to consume paths via `resolve_path.py` +- [x] Refactor `scripts/o3/o3_preprocess.py` (Python) to pull its execution map via `build_manifest.py` +- [x] Refactor `scripts/o3/o3_score.py` (Python or R) to pull its execution map via `build_manifest.py` + - [x] Verify full run locally + - [x] Verify full run remotely (dry-run or execution on S3 via `fsspec`/R cloud tools) + +## Phase 5: Prepare for and merge with `indicators` branch +- [ ] Migrate the remaining indicator configuration files (`pm25_config.json`, etc.) to the new schema +- [ ] Run the `test_manifest.py` harness against all newly migrated indicators to catch syntax/schema errors early +- [ ] Update execution scripts for all remaining indicators to consume the new `build_manifest` workflow +- [ ] change `xx_indictator.py` name to `xx_score.py` as each indicator's code is upgraded to data versioning +- [ ] Run final end-to-end testing across the entire pipeline universe +- [ ] Merge branch into `indicators` diff --git a/scripts/build_manifest_specification.md b/scripts/build_manifest_specification.md new file mode 100644 index 0000000..bcda186 --- /dev/null +++ b/scripts/build_manifest_specification.md @@ -0,0 +1,185 @@ +# Module Specification: build_manifest.py +**Purpose:** Provide a high-level orchestration planning tool shared across Python and R runtimes via reticulate. It reads either an indicator's or a shared asset's stage-specific configuration block, coordinates with `resolve_path.py`, and returns a single unified "Run Manifest" dictionary containing every file path required for that execution phase. + +--- + +## 1. System Requirements & Context +- **Language Compatibility:** Pure Python 3.x using only standard libraries (`json`, `os`, `pathlib`). No third-party dependencies. +- **Interoperability:** Must expose a flat functional interface at the module root level to support smooth R `reticulate` execution ergonomics. Under the hood, it must import and utilize `resolve_path.py` to obtain physical storage vectors. +- **Directory Context:** Assumes execution occurs from the `project_root/` directory. +- **Module Location:** The manifest module will live at `./scripts/shared/build_manifest.py`. + +--- + +## 2. Core Functional Requirements (The "What") + +The module acts as a workflow compiler. Instead of operational scripts parsing raw JSON trees to discover their inputs, this module processes the configuration schema to produce an actionable bill of materials. + +### 1. Configuration Target Structure +The module must support two target domains: + +- **Indicator target:** Read `./scripts/{name}/{name}_config.json`, navigate to `versions -> {version} -> stages -> {stage}`. +- **Shared target:** Read `./scripts/shared/shared_config.json`, navigate to `assets -> {name} -> {version} -> stages -> {stage}`. + +In both cases, the module must parse the targeted stage block's explicit `inputs` and `outputs` mappings. A stage may omit `inputs`; in that case, the returned manifest must contain an empty `inputs` dictionary. + +### 2. Dependency Tracking Invariant +When an entry under a stage's `inputs` section points to a global shared asset (e.g., `type: "shared_asset"`), `build_manifest.py` must automatically step back up the configuration tree, query `resolve_path.py` to identify the precise version string pinned under `required_shared_assets` for that indicator version, and then pass that information along to get the asset's shared paths. The calling processing script should never have to explicitly declare or look up dependency versions. + +Indicator-owned fetched assets are now modeled as outputs of the indicator's `fetch` stage. Shared downloaded assets are modeled as outputs of the shared asset's `fetch` stage. Shared preprocessed assets are modeled as outputs of the shared asset's `preprocess` stage. + +This dependency-version lookup rule applies to indicator targets. Shared targets are first-class manifest targets themselves and must be compilable directly from `shared_config.json` without being routed through an indicator config. + +### 3. Output Data Shape Contract +The function must compile and return a **single Python Dictionary** containing exactly two top-level keys: `"inputs"` and `"outputs"`. + +The value of every single nested asset identity key within this manifest must follow the resolver contract and include at least the keys shown below: +```python +{ + "root": "string base path", + "relative": "string versioned relative path (or template)" +} + +``` + +--- + +## 3. Required Module Interface (Exposed Function) + +### `get_stage_manifest(target_type: str, name: str, stage: str, version: str, environment: str = "local") -> dict` + +`target_type` must accept only `"indicator"` or `"shared"`, and invalid values must fail fast with a clear error. + +`environment` must accept only `"local"` or `"remote"`, and invalid values must fail fast with a clear error. + +#### Expected Behavior: + +1. **Load Configuration:** +- If `target_type="indicator"`, read and parse `./scripts/{name}/{name}_config.json`. +- If `target_type="shared"`, read and parse `./scripts/shared/shared_config.json`. +2. **Isolate Stage Requirements:** +- If `target_type="indicator"`, navigate to `["versions"][version]["stages"][stage]`. +- If `target_type="shared"`, navigate to `["assets"][name][version]["stages"][stage]`. + +If the target, stage, or version does not exist, raise an explicit descriptive error. +3. **Compile Inputs:** Loop through all items defined inside the stage's `inputs` block: +- For indicator targets: + - If an input maps to an indicator fetch asset, request its coordinates from `resolve_path.get_download_path()`, which resolves through the indicator's `fetch` stage outputs. + - If an input maps to a global shared asset, query `resolve_path.get_dependency_version()` to find the required version, then pass that version to `resolve_path.get_shared_asset_path()`. + - If an input maps to an indicator-owned `file`, `directory`, or `template`, compile it directly against the indicator root path using its `relative_path`. +- For shared targets: + - If the stage defines no `inputs`, return an empty `inputs` dictionary. + - If shared-stage inputs are later defined, they must be compiled strictly from their declared entry types. The implementation must not invent fallback rules for undeclared or transitional shared input shapes. + + +4. **Compile Outputs:** Loop through all items defined inside the stage's `outputs` block. Match them against indicator paths or directory structures, compiling them into the identical dual-key dictionary output format. +* For indicator targets, compile outputs against the indicator root path using `relative_path`. +* For shared targets, compile outputs against the shared root path using the output entry's declared relative path field. Template tokens such as `{postal}` or `{fips}` must remain intact. +5. **Consolidate and Return:** Return the unified manifest layout. + +--- + +## 4. Manifest Dictionary Structural Contract (Target Output Shape) + +The AI code generator must guarantee that calling this module results in a dictionary nested exactly like the sample structure below: + +```python +{ + "inputs": { + "primary_asset_name": { + "root": "./pipeline/o3/", + "relative": "v1.0/downloads/2020/2020_ozone_daily_8hour_maximum.txt.gz" + }, + "dependent_shared_asset_name": { + "root": "./pipeline/shared/", + "relative": "census_block_weights/1.0/preprocessed_input/census_block_weights_2020_{postal}.csv" + } + }, + "outputs": { + "output_directory": { + "root": "./pipeline/o3/", + "relative": "v1.0/preprocessed_input/" + }, + "main_tract_averages": { + "root": "./pipeline/o3/", + "relative": "v1.0/preprocessed_input/o3_tract_annual_average.csv" + } + } +} + +``` + +Shared-target example: + +```python +{ + "inputs": {}, + "outputs": { + "tiger_bg_2020": { + "root": "./pipeline/shared/", + "relative": "downloads/tiger_lines/2020/bg/tl_2020_{fips}_bg.zip" + } + } +} + +``` + +--- + +## 5. Implementation Verification / Test Cases + +1. **Local Preprocess Context Resolution:** +`get_stage_manifest(target_type="indicator", name="o3", stage="preprocess", version="1.0", environment="local")` +=> Must return an input dictionary mapping both internal indicator assets and external `census_block_weights` location dictionaries with local file-system roots. +2. **Remote Score Context Resolution:** +`get_stage_manifest(target_type="indicator", name="o3", stage="score", version="1.0", environment="remote")` +=> Must return an input/output mapping dictionary prefixing all `"root"` parameters with the explicit S3 protocol block retrieved from the configuration file, keeping relative string structures preserved for pipeline orchestration. +3. **Local Shared Fetch Context Resolution:** +`get_stage_manifest(target_type="shared", name="tiger_bg", stage="fetch", version="2020", environment="local")` +=> Must return an outputs dictionary rooted at the shared local path with the unexpanded state-scoped relative template preserved. +4. **Local Shared Preprocess Context Resolution:** +`get_stage_manifest(target_type="shared", name="census_block_weights", stage="preprocess", version="1.0", environment="local")` +=> Must return the currently declared preprocess outputs for that shared asset with the shared local root and the relative template preserved. + +### Command-Line Dry-Run Validation Set + + +Assuming the current working directory is `scripts/`, the following harness commands should validate the current implementation surface: + +```bash +python3 test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0 +``` + +```bash +python3 test_harness/test_build_manifest.py --storage-mode remote --target-type indicator --name o3 --stage score --version 1.0 +``` + +```bash +python3 test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage fetch --version 1.0 +``` + +```bash +python3 test_harness/test_build_manifest.py --storage-mode local --target-type shared --name tiger_bg --stage fetch --version 2020 +``` + +```bash +python3 test_harness/test_build_manifest.py --storage-mode local --target-type shared --name census_block_weights --stage preprocess --version 1.0 +``` + +Negative fast-fail checks: + +```bash +python3 test_harness/test_build_manifest.py --storage-mode staging --target-type indicator --name o3 --stage preprocess --version 1.0 +``` + +```bash +python3 test_harness/test_build_manifest.py --storage-mode local --target-type nonsense --name o3 --stage preprocess --version 1.0 +``` + +## 6. Current Strictness Decision + +`build_manifest.py` should remain strict and should not implement fallback logic for incomplete or transitional config shapes. + +Shared assets are first-class manifest targets and must be supported directly by the manifest interface. + +`census_block_weights` is still an evolving shared-asset configuration surface. Until its final stage layout is settled, the manifest implementation should only honor the stages and fields explicitly present in config, without adding compatibility layers or silent fallbacks. diff --git a/scripts/o3/o3_config.json b/scripts/o3/o3_config.json index 8e94841..7d24933 100644 --- a/scripts/o3/o3_config.json +++ b/scripts/o3/o3_config.json @@ -1,14 +1,184 @@ { - "local_root_path": "./pipeline/", - "remote_root_path": "s3://pedp-data-preserved/ejscreen-data-processing/o3/pipeline/", - "preprocessed_tract_output_relative_path": "preprocessed_input/o3_tract_annual_average.csv", - "indicator_output_relative_path_template": "output/indicators/{postal}", - "downloads": { + "local_root_path": "./pipeline/o3/", + "remote_root_path": "s3://pedp-data-preserved/ejscreen-data-processing/pipeline/o3/", + "download_settings": { "request_timeout_seconds": 120, - "chunk_size_bytes": 1048576, - "raw_o3_2020": { - "relative_path": "downloads/2020/2020_ozone_daily_8hour_maximum.txt.gz", - "source_url": "https://ofmpub.epa.gov/rsig/rsigserver?data/FAQSD/outputs/2020_ozone_daily_8hour_maximum.txt.gz" + "chunk_size_bytes": 1048576 + }, + "versions": { + "1.0": { + "required_shared_assets": { + "census_block_weights": "1.0" + }, + "notes": "This is the version where the scoring stage will use EmmaLi's new-improved census block weight file that includes the acs_2022_bg_pop column to use for identifying zero-population block groups.", + "stages": { + "fetch": { + "outputs": { + "raw_o3": { + "type": "download", + "relative_path": "v1.0/downloads/2020/2020_ozone_daily_8hour_maximum.txt.gz", + "source_url": "https://ofmpub.epa.gov/rsig/rsigserver?data/FAQSD/outputs/2020_ozone_daily_8hour_maximum.txt.gz" + } + } + }, + "preprocess": { + "inputs": { + "primary_o3": { + "type": "indicator_download", + "key": "raw_o3" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v1.0/preprocessed_input/" + }, + "main_tract_averages": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/o3_tract_annual_average.csv" + } + } + }, + "score": { + "inputs": { + "preprocessed_tracts": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/o3_tract_annual_average.csv" + }, + "weights": { + "type": "shared_asset", + "key": "census_block_weights", + "category": "preprocessed_input" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v1.0/output/indicators/" + }, + "indicator_output_template": { + "type": "template", + "relative_path": "v1.0/output/indicators/{postal}/" + } + } + } + } + }, + "0.6": { + "required_shared_assets": { + "census_block_weights": "0.6" + }, + "notes": "This is the version where the scoring stage will read in the whole-country census block weight file and then select only the state data it needs for each run", + "stages": { + "fetch": { + "outputs": { + "raw_o3": { + "type": "download", + "relative_path": "v1.0/downloads/2020/2020_ozone_daily_8hour_maximum.txt.gz", + "source_url": "https://ofmpub.epa.gov/rsig/rsigserver?data/FAQSD/outputs/2020_ozone_daily_8hour_maximum.txt.gz" + } + } + }, + "preprocess": { + "inputs": { + "primary_o3": { + "type": "indicator_download", + "key": "raw_o3" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v1.0/preprocessed_input/" + }, + "main_tract_averages": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/o3_tract_annual_average.csv" + } + } + }, + "score": { + "inputs": { + "preprocessed_tracts": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/o3_tract_annual_average.csv" + }, + "weights": { + "type": "shared_asset", + "key": "census_block_weights", + "category": "preprocessed_input" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v0.6/output/indicators/" + }, + "indicator_output_template": { + "type": "template", + "relative_path": "v0.6/output/indicators/{postal}/" + } + } + } + } + }, + "0.5": { + "required_shared_assets": { + "census_block_weights": "0.5" + }, + "notes": "This was version 1.0 but I've backed down the version # as I work on issue #41 and change how the scoring code reads census_block_weights data. The fetch and preprocess stages are identical to 1.0 intentionally. Only the output folder for the scoring stage has been changed to 0.5", + "stages": { + "fetch": { + "outputs": { + "raw_o3": { + "type": "download", + "relative_path": "v1.0/downloads/2020/2020_ozone_daily_8hour_maximum.txt.gz", + "source_url": "https://ofmpub.epa.gov/rsig/rsigserver?data/FAQSD/outputs/2020_ozone_daily_8hour_maximum.txt.gz" + } + } + }, + "preprocess": { + "inputs": { + "primary_o3": { + "type": "indicator_download", + "key": "raw_o3" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v1.0/preprocessed_input/" + }, + "main_tract_averages": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/o3_tract_annual_average.csv" + } + } + }, + "score": { + "inputs": { + "preprocessed_tracts": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/o3_tract_annual_average.csv" + }, + "weights": { + "type": "shared_asset", + "key": "census_block_weights", + "category": "preprocessed_input" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v0.5/output/indicators/" + }, + "indicator_output_template": { + "type": "template", + "relative_path": "v0.5/output/indicators/{postal}/" + } + } + } + } } } -} \ No newline at end of file +} diff --git a/scripts/o3/o3_config.py b/scripts/o3/o3_config.py deleted file mode 100644 index c91453c..0000000 --- a/scripts/o3/o3_config.py +++ /dev/null @@ -1,142 +0,0 @@ -"""o3_config.py - -Purpose: - Load and validate the O3 pipeline configuration shared by the fetch, - preprocess, and indicator scripts. - -Configuration summary: - - Reads scripts/o3/o3_config.json. - - Validates the local and remote root paths. - - Validates the tract-level preprocess output path and per-state indicator - output template. - - Validates the nested raw-download entry for the 2020 national O3 - source file. - -Public helpers: - - get_o3_config(): returns one validated o3Config instance and caches it. - - resolve_local_o3_root_path(): resolves the configured local pipeline root - relative to the scripts/o3 folder. -""" - -from __future__ import annotations - -from dataclasses import dataclass -from functools import lru_cache -from pathlib import Path -import json - - -O3_CONFIG_PATH = Path(__file__).with_name('o3_config.json') -RAW_O3_2020_DOWNLOAD_KEY = 'raw_o3_2020' -REQUIRED_CONFIG_FIELDS = ( - 'local_root_path', - 'remote_root_path', - 'preprocessed_tract_output_relative_path', - 'indicator_output_relative_path_template', - 'downloads', -) - -REQUIRED_DOWNLOAD_FIELDS = ( - 'relative_path', - 'source_url', -) - - -@dataclass(frozen=True, slots=True) -class O3Config: - local_root_path: str - remote_root_path: str - preprocessed_tract_output_relative_path: str - indicator_output_relative_path_template: str - raw_download_relative_path: str - source_url: str - request_timeout_seconds: int - chunk_size_bytes: int - - -def _require_string_field(raw_config: dict[str, object], field_name: str) -> str: - value = raw_config.get(field_name) - if not isinstance(value, str) or not value.strip(): - raise ValueError(f'{O3_CONFIG_PATH.name} field {field_name!r} must be a non-empty string') - return value.strip() - - -def _require_int_field(raw_config: dict[str, object], field_name: str) -> int: - value = raw_config.get(field_name) - if not isinstance(value, int) or value <= 0: - raise ValueError(f'{O3_CONFIG_PATH.name} field {field_name!r} must be a positive integer') - return value - - -def _validate_relative_path(field_name: str, relative_path: str) -> None: - if relative_path.startswith('/') or relative_path.startswith('s3://'): - raise ValueError(f'{O3_CONFIG_PATH.name} field {field_name!r} must be a relative path') - - -def _require_dict_field(raw_config: dict[str, object], field_name: str) -> dict[str, object]: - value = raw_config.get(field_name) - if not isinstance(value, dict): - raise ValueError(f'{O3_CONFIG_PATH.name} field {field_name!r} must be a JSON object') - return value - - -@lru_cache(maxsize=1) -def get_o3_config() -> O3Config: - """Return the validated O3 configuration from o3_config.json.""" - if not O3_CONFIG_PATH.exists(): - raise FileNotFoundError(f'O3 config not found: {O3_CONFIG_PATH}') - - with O3_CONFIG_PATH.open('r', encoding='utf-8') as handle: - raw_config = json.load(handle) - - if not isinstance(raw_config, dict): - raise ValueError(f'{O3_CONFIG_PATH.name} must contain a JSON object') - - missing_fields = [field_name for field_name in REQUIRED_CONFIG_FIELDS if field_name not in raw_config] - if missing_fields: - raise ValueError(f'{O3_CONFIG_PATH.name} missing required fields: {", ".join(missing_fields)}') - - downloads_config = _require_dict_field(raw_config, 'downloads') - download_settings_missing = [ - field_name - for field_name in ('request_timeout_seconds', 'chunk_size_bytes', RAW_O3_2020_DOWNLOAD_KEY) - if field_name not in downloads_config - ] - if download_settings_missing: - raise ValueError( - f'{O3_CONFIG_PATH.name} downloads missing required fields: {", ".join(download_settings_missing)}' - ) - - raw_o3_download_config = _require_dict_field(downloads_config, RAW_O3_2020_DOWNLOAD_KEY) - raw_download_missing_fields = [ - field_name for field_name in REQUIRED_DOWNLOAD_FIELDS if field_name not in raw_o3_download_config - ] - if raw_download_missing_fields: - raise ValueError( - f'{O3_CONFIG_PATH.name} downloads.{RAW_O3_2020_DOWNLOAD_KEY} missing required fields: ' - f'{", ".join(raw_download_missing_fields)}' - ) - - config = O3Config( - local_root_path=_require_string_field(raw_config, 'local_root_path'), - remote_root_path=_require_string_field(raw_config, 'remote_root_path'), - preprocessed_tract_output_relative_path=_require_string_field(raw_config, 'preprocessed_tract_output_relative_path'), - indicator_output_relative_path_template=_require_string_field(raw_config, 'indicator_output_relative_path_template'), - raw_download_relative_path=_require_string_field(raw_o3_download_config, 'relative_path'), - source_url=_require_string_field(raw_o3_download_config, 'source_url'), - request_timeout_seconds=_require_int_field(downloads_config, 'request_timeout_seconds'), - chunk_size_bytes=_require_int_field(downloads_config, 'chunk_size_bytes'), - ) - - _validate_relative_path('raw_download_relative_path', config.raw_download_relative_path) - _validate_relative_path('preprocessed_tract_output_relative_path', config.preprocessed_tract_output_relative_path) - _validate_relative_path('indicator_output_relative_path_template', config.indicator_output_relative_path_template) - if not config.source_url.lower().startswith(('http://', 'https://')): - raise ValueError(f'{O3_CONFIG_PATH.name} field \"source_url\" must be an http(s) URL') - return config - - -def resolve_local_o3_root_path(o3_dir: Path) -> str: - """Resolve the configured local O3 pipeline root from the o3 script directory.""" - config = get_o3_config() - return str((o3_dir / config.local_root_path).resolve()) \ No newline at end of file diff --git a/scripts/o3/o3_preprocess.py b/scripts/o3/o3_preprocess.py index 798fbcf..cf3afde 100644 --- a/scripts/o3/o3_preprocess.py +++ b/scripts/o3/o3_preprocess.py @@ -6,20 +6,33 @@ the tract-level intermediate CSV used by the indicator step. Process summary: - - Resolve local or remote Ozone storage from o3_config.json. + - Resolve local or remote Ozone storage from the manifest + resolver. - Validate the compressed file header before performing the full read. - Read only the required columns from the raw source. - Validate tract GEOIDs, dates, and numeric Ozone values. - Aggregate to one annual average concentration per tract. - Write the tract-level CSV and log the output range. -Runtime arguments: - - storage_mode - Required. Either local or remote. +Runtime arguments (current defaults shown): + - storage_mode (positional): one of `local` or `remote`. + - -v/--version: optional config version to use. Default: `1.0`. + - --dry-run: long-only flag. When present the script validates the preprocess manifest + and source file headers, then exits without reading or writing full outputs. Outputs: - - preprocessed_input/o3_tract_annual_average.csv under the active Ozone root. - - o3_preprocess.log in scripts/o3. + - v1.0/preprocessed_input/o3_tract_annual_average.csv under the active Ozone root (versioned by manifest). + - o3_preprocess.log in scripts/o3. + +Examples (run from the `scripts` folder): + - Dry-run (local): + python3 o3/o3_preprocess.py local --dry-run + + - Full run (local, explicit version): + python3 o3/o3_preprocess.py local -v 1.0 + + - Full run (remote): + python3 o3/o3_preprocess.py remote -v 1.0 + """ from __future__ import annotations @@ -32,8 +45,16 @@ from pathlib import Path import pandas as pd +import sys +# Ensure the `scripts/shared` folder is importable so modules in that +# folder (e.g. build_manifest.py, resolve_path.py) can be imported as +# top-level modules when running this script directly. +SHARED_SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "shared" +if str(SHARED_SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SHARED_SCRIPTS_DIR)) -from o3_config import get_o3_config, resolve_local_o3_root_path +import build_manifest +import resolve_path O3_DIR = Path(__file__).resolve().parent @@ -44,19 +65,29 @@ TRACT_GEOID_COLUMN = 'tract_geoid' ANNUAL_AVERAGE_COLUMN = 'annual_average_ten_highest_MDA8' +# Pre-compile or centralize frequently used regexes/patterns as module +# constants to avoid recreating them on each function call. +FIPS_PATTERN = r'\d{11}' + @dataclass(frozen=True, slots=True) class Config: storage_mode: str + version: str local_root_path: str remote_root_path: str raw_download_relative_path: str preprocessed_tract_output_relative_path: str + dry_run: bool = False def get_config(argv=None) -> Config: - """Parse runtime arguments and merge them with validated Ozone config values.""" - raw_config = get_o3_config() + """Parse runtime arguments and derive canonical paths from the manifest/resolver. + + This function uses `build_manifest.get_stage_manifest()` (preprocess stage) + and `resolve_path.get_indicator_root()` to populate the runtime config. + No compatibility fallback to the old flat JSON format is provided. + """ parser = argparse.ArgumentParser( description='Read the raw national Ozone .txt.gz file, compute tract-level annual average concentration, and write the preprocessed tract CSV.' @@ -66,14 +97,63 @@ def get_config(argv=None) -> Config: choices=('local', 'remote'), help='Select whether the script reads and writes through the configured local root path or remote S3 root path.', ) + parser.add_argument( + '-v', '--version', + dest='version', + default='1.0', + help='Optional: config version to base processing on (current default: 1.0)' + ) + # Long-only dry-run flag (no short alias) + parser.add_argument( + '--dry-run', + dest='dry_run', + action='store_true', + help='Dry run: validate manifest and headers but do not process or write outputs.', + ) args = parser.parse_args(argv) + # Build the preprocess manifest for this indicator/version. We expect the + # preprocess stage to define an input `primary_o3` (indicator download) and + # an output `main_tract_averages` (the tract CSV). + manifest = build_manifest.get_stage_manifest( + target_type='indicator', + name='o3', + stage='preprocess', + version=args.version, + environment=args.storage_mode, + ) + + inputs = manifest.get('inputs', {}) + outputs = manifest.get('outputs', {}) + + if 'primary_o3' not in inputs: + raise RuntimeError('Preprocess manifest missing required input: primary_o3') + if 'main_tract_averages' not in outputs: + raise RuntimeError('Preprocess manifest missing required output: main_tract_averages') + + # Manifest entries may already include a compiled "root" and "relative". + raw_entry = inputs['primary_o3'] + preproc_entry = outputs['main_tract_averages'] + + raw_rel = raw_entry.get('relative') + preproc_rel = preproc_entry.get('relative') + + if not raw_rel or not isinstance(raw_rel, str): + raise RuntimeError('Invalid relative path for primary_o3 in preprocess manifest') + if not preproc_rel or not isinstance(preproc_rel, str): + raise RuntimeError('Invalid relative path for main_tract_averages in preprocess manifest') + + local_root = resolve_path.get_indicator_root('o3', args.version, 'local') + remote_root = resolve_path.get_indicator_root('o3', args.version, 'remote') + return Config( storage_mode=args.storage_mode, - local_root_path=resolve_local_o3_root_path(O3_DIR), - remote_root_path=raw_config.remote_root_path, - raw_download_relative_path=raw_config.raw_download_relative_path, - preprocessed_tract_output_relative_path=raw_config.preprocessed_tract_output_relative_path, + version=args.version, + local_root_path=local_root, + remote_root_path=remote_root, + raw_download_relative_path=raw_rel, + preprocessed_tract_output_relative_path=preproc_rel, + dry_run=args.dry_run, ) @@ -198,7 +278,7 @@ def validate_and_prepare_raw_table(df: pd.DataFrame) -> pd.DataFrame: require_columns(df, (RAW_FIPS_COLUMN, RAW_DATE_COLUMN, RAW_O3_COLUMN)) prepared = df[[RAW_FIPS_COLUMN, RAW_DATE_COLUMN, RAW_O3_COLUMN]].copy() prepared[RAW_FIPS_COLUMN] = prepared[RAW_FIPS_COLUMN].astype('string').str.strip() - invalid_fips_mask = prepared[RAW_FIPS_COLUMN].isna() | ~prepared[RAW_FIPS_COLUMN].str.fullmatch(r'\d{11}') + invalid_fips_mask = prepared[RAW_FIPS_COLUMN].isna() | ~prepared[RAW_FIPS_COLUMN].str.fullmatch(FIPS_PATTERN) if invalid_fips_mask.any(): invalid_samples = prepared.loc[invalid_fips_mask, RAW_FIPS_COLUMN].drop_duplicates().astype(str).head(5).tolist() raise RuntimeError( @@ -271,6 +351,18 @@ def main(argv=None) -> int: header_df = read_raw_o3_header(raw_input_path) require_columns(header_df, (RAW_FIPS_COLUMN, RAW_DATE_COLUMN, RAW_O3_COLUMN)) logging.info('Raw Ozone header validation passed') + # If dry-run was requested, we've validated manifest resolution and the + # raw file header; exit early without performing the full read/aggregation + # or writing outputs. + if cfg.dry_run: + logging.info('Dry run enabled; skipping full read/processing and write.') + logging.info('DRY RUN: manifest and header validated.') + logging.info('raw_input_path=%s', raw_input_path) + logging.info('output_path=%s', output_path) + print('DRY RUN: manifest and header validated.') + print(f'raw_input_path={raw_input_path}') + print(f'output_path={output_path}') + return 0 logging.info('Reading raw Ozone input table') raw_df = read_raw_o3_table(raw_input_path) @@ -283,4 +375,4 @@ def main(argv=None) -> int: if __name__ == '__main__': - raise SystemExit(main()) \ No newline at end of file + raise SystemExit(main()) diff --git a/scripts/o3/o3_indicator.py b/scripts/o3/o3_score.py similarity index 54% rename from scripts/o3/o3_indicator.py rename to scripts/o3/o3_score.py index 393d208..0114337 100644 --- a/scripts/o3/o3_indicator.py +++ b/scripts/o3/o3_score.py @@ -1,10 +1,14 @@ -"""o3_indicator.py +"""o3_score.py Purpose: Read tract-level Ozone averages, expand them to block groups with the shared census block weights inputs, apply the zero-population null rule, and write per-state final_bg_scores.csv outputs. +Notes: + - The final block-group output filename is fixed to ``final_bg_scores.csv`` + and is not configurable via the CLI. + Process summary: - Resolve the Ozone and shared roots for local or remote mode. - Read the tract-level preprocess output. @@ -14,43 +18,56 @@ - Set o3_score to null for zero-population block groups. - Write per-state final_bg_scores.csv files and state summary logs. -Runtime arguments: - - storage_mode - Required. Either local or remote. - - --state - Optional two-letter state filter. If omitted, process all configured states. +Runtime arguments (current defaults shown): + - -l/--location: required one of `local` or `remote`. + - --state: required two-letter state code (e.g. `WY`) or `all` (case-insensitive) to process the full + configured set. Use `--state all` to iterate across all configured states (the code maps this to + an internal `None` which `load_state_targets()` interprets as "all"). + - --output-dir: optional explicit output directory (local path or S3 URI). Default: none (uses indicator output template). + - -v/--version: optional config version to use. Default: `1.0`. + - --dry-run: long-only flag. When present the script validates manifests and prints resolved paths + without performing any I/O. Outputs: - - output/indicators/{postal}/final_bg_scores.csv under the active Ozone root. - - o3_indicator.log in scripts/o3. -""" + - output/indicators/{postal}/final_bg_scores.csv under the active Ozone root (filename is fixed to + `final_bg_scores.csv`). + - o3_score.log in scripts/o3. + +Examples (run from the `scripts` folder): + - Dry-run for Wyoming (local): + python3 o3/o3_score.py --location local --state WY --dry-run + + - Full run for all configured states (local): + python3 o3/o3_score.py --location local --state all + - Full run for a specific version (remote): + python3 o3/o3_score.py --location remote -v 1.0 --state CA + +""" + from __future__ import annotations import argparse -from dataclasses import dataclass +from dataclasses import dataclass, replace from datetime import datetime +from decimal import Decimal, InvalidOperation import importlib -import importlib.util -import json import logging from pathlib import Path import sys -from typing import Any import pandas as pd -from o3_config import get_o3_config, resolve_local_o3_root_path - O3_DIR = Path(__file__).resolve().parent -DEFAULT_LOG_FILENAME = 'o3_indicator.log' +DEFAULT_LOG_FILENAME = 'o3_score.log' DEFAULT_FINAL_BG_SCORES_FILENAME = 'final_bg_scores.csv' TRACT_GEOID_COLUMN = 'tract_geoid' ANNUAL_AVERAGE_COLUMN = 'annual_average_ten_highest_MDA8' FINAL_SCORE_COLUMN = 'o3_score' #Edit this to match EJAM/EJSCREEN e.g. o3 BLOCK_GROUP_GEOID_COLUMN = 'block_group_geoid' BLOCK_GROUP_POP_COLUMN = 'block_group_pop' +BLOCK_GROUP_ACS_2022_POP_COLUMN = 'acs_2022_bg_pop' O3_STORAGE_MODES = ('local', 'remote') @@ -66,6 +83,23 @@ def _resolve_scripts_dir() -> Path: if str(SCRIPTS_DIR) not in sys.path: sys.path.insert(0, str(SCRIPTS_DIR)) +# Ensure the `scripts/shared` folder is importable so modules in that +# folder (e.g. build_manifest.py, resolve_path.py) can be imported as +# top-level modules when running this script directly. +SHARED_SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "shared" +if str(SHARED_SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SHARED_SCRIPTS_DIR)) + + +def parse_version_decimal(version_str: str) -> Decimal: + try: + return Decimal(str(version_str)) + except (InvalidOperation, TypeError) as exc: + raise RuntimeError(f'Invalid version string: {version_str}') from exc + +import build_manifest +import resolve_path + try: from shared.state_config import StateConfig, get_state_config, get_state_config_list except Exception as exc: @@ -74,21 +108,16 @@ def _resolve_scripts_dir() -> Path: 'that the repository root (containing the scripts directory) is on PYTHONPATH.' ) from exc -try: - from shared.shared_config import get_shared_config, resolve_local_shared_root_path -except Exception as exc: - raise RuntimeError( - 'Failed to import shared.shared_config. Ensure scripts/shared/shared_config.py is present and ' - 'that the repository root (containing the scripts directory) is on PYTHONPATH.' - ) from exc @dataclass(frozen=True, slots=True) class Config: storage_mode: str state: str | None + version: str = '1.0' + version_decimal: Decimal | None = None output_dir: str | None = None - final_bg_scores_filename: str = DEFAULT_FINAL_BG_SCORES_FILENAME + dry_run: bool = False @dataclass(frozen=True, slots=True) @@ -109,15 +138,17 @@ def get_config(argv=None) -> Config: description='Read tract-level Ozone scores and produce per-state block-group final scores.' ) parser.add_argument( - 'storage_mode', + '-l', '--location', + dest='storage_mode', choices=O3_STORAGE_MODES, + required=True, help='Select whether the script reads and writes through the local root path or remote S3 root path.', ) parser.add_argument( - '--state', + '-s', '--state', dest='state', - default=defaults.state, - help='Optional two-letter state code to process a single state.', + required=True, + help="Specify a two-letter state code (e.g. 'WY') or 'all' to process the full configured set.", ) parser.add_argument( '--output-dir', @@ -125,23 +156,36 @@ def get_config(argv=None) -> Config: default=defaults.output_dir, help='Optional explicit local path or S3 URI for the state output directory.', ) + + parser.add_argument( + '-v', '--version', + dest='version', + default=defaults.version, + help='Optional: config version to base processing on (current default: 1.0)' + ) + # Long-only dry-run flag (no short alias) parser.add_argument( - '--final-bg-scores-filename', - dest='final_bg_scores_filename', - default=defaults.final_bg_scores_filename, - help=f'Output filename for final state scores (default: {defaults.final_bg_scores_filename})', + '--dry-run', + dest='dry_run', + action='store_true', + help='Dry run: validate manifest and paths but do not read or write any files.', ) args = parser.parse_args(argv) + # Accept explicit 'all' (case-insensitive) to indicate processing the full set. state = None if args.state: - state = normalize_state_code(args.state) + if isinstance(args.state, str) and args.state.strip().lower() == 'all': + state = None # This value gets interpreted by load_state_targets as "load all states" + else: + state = normalize_state_code(args.state) return Config( storage_mode=args.storage_mode, state=state, + version=args.version, output_dir=args.output_dir, - final_bg_scores_filename=args.final_bg_scores_filename, + dry_run=bool(args.dry_run), ) @@ -240,55 +284,52 @@ def write_df_s3_or_local(df: pd.DataFrame, out_path: str) -> None: df.to_csv(out_path, index=False) -def get_active_o3_root_path(storage_mode: str) -> str: - if storage_mode == 'local': - return resolve_local_o3_root_path(O3_DIR) - if storage_mode == 'remote': - return get_o3_config().remote_root_path - raise ValueError(f'Unsupported storage mode: {storage_mode}') -def get_active_shared_root_path(storage_mode: str) -> str: - if storage_mode == 'local': - # 12 May 2026, not adopting Eric's change for this chunk of - # code. I don't think that hard-coding "shared" here is or should - # be needed. - return resolve_local_shared_root_path(SCRIPTS_DIR) - if storage_mode == 'remote': - return get_shared_config().remote_root_path - raise ValueError(f'Unsupported storage mode: {storage_mode}') +def resolve_paths(cfg: Config, state_config: StateConfig, manifest: dict) -> ResolvedPaths: + """Resolve the tract input, shared block-weight input, and state output paths. + The manifest must be provided by the caller to avoid duplicate manifest + lookups; callers should request the stage manifest once and pass it through. + """ + inputs = manifest.get('inputs', {}) + outputs = manifest.get('outputs', {}) -def resolve_paths(cfg: Config, state_config: StateConfig) -> ResolvedPaths: - """Resolve the tract input, shared block-weight input, and state output paths.""" - o3_config = get_o3_config() - shared_cfg = get_shared_config() - o3_root_path = get_active_o3_root_path(cfg.storage_mode) - shared_root_path = get_active_shared_root_path(cfg.storage_mode) + # preprocessed_tracts should be a plain file input (relative to indicator root) + tract_entry = inputs.get('preprocessed_tracts') + if not tract_entry: + raise RuntimeError('Score manifest missing required input: preprocessed_tracts') + indicator_root = resolve_path.get_indicator_root('o3', cfg.version, cfg.storage_mode) + tract_scores_path = join_root_and_relative_path(indicator_root, tract_entry['relative']) - tract_scores_path = join_root_and_relative_path( - o3_root_path, - o3_config.preprocessed_tract_output_relative_path, - ) - census_block_weights_relative_path = shared_cfg.census_block_weights_relative_path_template.format( - postal=state_config.postal, - ) + # weights is a shared asset; resolve shared version and root via resolver + weights_entry = inputs.get('weights') + if not weights_entry: + raise RuntimeError('Score manifest missing required input: weights') + shared_version = resolve_path.get_dependency_version('o3', cfg.version, 'census_block_weights') + shared_root = resolve_path.get_shared_root('census_block_weights', shared_version, cfg.storage_mode) census_block_weights_path = join_root_and_relative_path( - shared_root_path, - census_block_weights_relative_path, + shared_root, + weights_entry['relative'].format(postal=state_config.postal), ) + + # output directory/template (indicator outputs are relative to indicator root) + output_dir_entry = outputs.get('indicator_output_template') + if not output_dir_entry: + raise RuntimeError('Score manifest missing required output: indicator_output_template') output_dir = cfg.output_dir or join_root_and_relative_path( - o3_root_path, - o3_config.indicator_output_relative_path_template.format(postal=state_config.postal), + indicator_root, + output_dir_entry['relative'].format(postal=state_config.postal), ) + return ResolvedPaths( state_config=state_config, - o3_root_path=o3_root_path, - shared_root_path=shared_root_path, + o3_root_path=indicator_root, + shared_root_path=shared_root, tract_scores_path=tract_scores_path, census_block_weights_path=census_block_weights_path, output_dir=output_dir, - final_bg_scores_path=join_path_and_file(output_dir, cfg.final_bg_scores_filename), + final_bg_scores_path=join_path_and_file(output_dir, DEFAULT_FINAL_BG_SCORES_FILENAME), ) @@ -319,7 +360,7 @@ def prepare_tract_scores(df: pd.DataFrame) -> pd.DataFrame: return prepared.sort_values(TRACT_GEOID_COLUMN).reset_index(drop=True) -def prepare_block_group_population(df: pd.DataFrame) -> pd.DataFrame: +def prepare_block_group_population_v0(df: pd.DataFrame) -> pd.DataFrame: """Reduce the shared block-weight input to one population row per block group.""" require_columns(df, (BLOCK_GROUP_GEOID_COLUMN, BLOCK_GROUP_POP_COLUMN), 'Census block weights CSV') prepared = df[[BLOCK_GROUP_GEOID_COLUMN, BLOCK_GROUP_POP_COLUMN]].copy() @@ -348,10 +389,59 @@ def prepare_block_group_population(df: pd.DataFrame) -> pd.DataFrame: return group_population.sort_values(BLOCK_GROUP_GEOID_COLUMN).reset_index(drop=True) -def build_final_scores(tract_scores: pd.DataFrame, block_group_population: pd.DataFrame) -> pd.DataFrame: +def prepare_block_group_population_v1(df: pd.DataFrame) -> pd.DataFrame: + """Prepare block-group population for version 1.0+: require acs_2022_bg_pop. + + Returns a DataFrame containing at minimum the block group geoid, the + existing `block_group_pop` (if present) and the `acs_2022_bg_pop` column, + plus the derived tract geoid. + """ + required = (BLOCK_GROUP_GEOID_COLUMN, BLOCK_GROUP_ACS_2022_POP_COLUMN, BLOCK_GROUP_POP_COLUMN) + require_columns(df, required, 'Census block weights CSV (v1)') + + prepared = df[[BLOCK_GROUP_GEOID_COLUMN, BLOCK_GROUP_POP_COLUMN, BLOCK_GROUP_ACS_2022_POP_COLUMN]].copy() + prepared[BLOCK_GROUP_GEOID_COLUMN] = prepared[BLOCK_GROUP_GEOID_COLUMN].astype('string').str.strip() + invalid_bg_mask = prepared[BLOCK_GROUP_GEOID_COLUMN].isna() | ~prepared[BLOCK_GROUP_GEOID_COLUMN].str.fullmatch(r'\d{12}') + if invalid_bg_mask.any(): + invalid_samples = prepared.loc[invalid_bg_mask, BLOCK_GROUP_GEOID_COLUMN].drop_duplicates().astype(str).head(5).tolist() + raise RuntimeError(f'Census block weights CSV contains invalid block group GEOIDs. Sample invalid values: {invalid_samples}') + + prepared[BLOCK_GROUP_POP_COLUMN] = pd.to_numeric(prepared[BLOCK_GROUP_POP_COLUMN], errors='raise') + if prepared[BLOCK_GROUP_POP_COLUMN].isna().any(): + raise RuntimeError('Census block weights CSV contains null block_group_pop values') + if (prepared[BLOCK_GROUP_POP_COLUMN] < 0).any(): + raise RuntimeError('Census block weights CSV contains negative block_group_pop values') + + # Validate ACS population column + prepared[BLOCK_GROUP_ACS_2022_POP_COLUMN] = pd.to_numeric(prepared[BLOCK_GROUP_ACS_2022_POP_COLUMN], errors='raise') + if prepared[BLOCK_GROUP_ACS_2022_POP_COLUMN].isna().any(): + raise RuntimeError('Census block weights CSV contains null BLOCK_GROUP_ACS_2022_POP_COLUMN values') + if (prepared[BLOCK_GROUP_ACS_2022_POP_COLUMN] < 0).any(): + raise RuntimeError('Census block weights CSV contains negative BLOCK_GROUP_ACS_2022_POP_COLUMN values') + + population_variants = prepared.groupby(BLOCK_GROUP_GEOID_COLUMN, dropna=False)[BLOCK_GROUP_POP_COLUMN].nunique(dropna=False) + inconsistent_geoids = population_variants[population_variants > 1].index.tolist() + if inconsistent_geoids: + raise RuntimeError( + 'Census block weights CSV contains inconsistent block_group_pop values for some block groups. ' + f'Sample GEOIDs: {inconsistent_geoids[:5]}' + ) + + group_population = prepared.drop_duplicates(subset=[BLOCK_GROUP_GEOID_COLUMN]).copy() + group_population[TRACT_GEOID_COLUMN] = group_population[BLOCK_GROUP_GEOID_COLUMN].str.slice(0, 11) + return group_population.sort_values(BLOCK_GROUP_GEOID_COLUMN).reset_index(drop=True) + + +def build_final_scores(tract_scores: pd.DataFrame, block_group_population: pd.DataFrame, version: Decimal | None = None) -> pd.DataFrame: """Join tract scores to block groups and apply the zero-population null rule.""" merged = block_group_population.merge(tract_scores, on=TRACT_GEOID_COLUMN, how='left') - positive_population_mask = merged[BLOCK_GROUP_POP_COLUMN] > 0 + # As of version 1.0, we use ACS-based population for enforcing the zero population rule, + # so we need that column from the input block group weights file. + if version >= Decimal('1.0'): + pop_col = BLOCK_GROUP_ACS_2022_POP_COLUMN + else: + pop_col = BLOCK_GROUP_POP_COLUMN + positive_population_mask = merged[pop_col] > 0 missing_positive_mask = positive_population_mask & merged[ANNUAL_AVERAGE_COLUMN].isna() if missing_positive_mask.any(): missing_samples = merged.loc[missing_positive_mask, BLOCK_GROUP_GEOID_COLUMN].astype(str).head(5).tolist() @@ -408,13 +498,47 @@ def log_state_summary( ) -def process_state(cfg: Config, state_config: StateConfig, tract_scores: pd.DataFrame) -> None: +def process_state(cfg: Config, state_config: StateConfig, tract_scores: pd.DataFrame, manifest: dict) -> None: """Build and write one state's final Ozone block-group score file.""" - paths = resolve_paths(cfg, state_config) + paths = resolve_paths(cfg, state_config, manifest) log_resolved_paths(paths, cfg) - block_weights_df = read_csv_s3_or_local(paths.census_block_weights_path, dtype={BLOCK_GROUP_GEOID_COLUMN: 'string'}) - block_group_population = prepare_block_group_population(block_weights_df) - final_scores = build_final_scores(tract_scores, block_group_population) + + # Note that some code below was changed for processing our v0.6 configuration, where + # we use only the whole-nation census block weight csv. But, because + # the column names did not change, this code works equally well for the v0.5 configuration, + # where the input file is one state at a time. + usecols = ['state_abb', BLOCK_GROUP_GEOID_COLUMN, BLOCK_GROUP_POP_COLUMN] + + # As of version 1.0, we require the ACS 2022 population column to be present in the block weights CSV. + if cfg.version_decimal >= Decimal('1.0'): + usecols.append(BLOCK_GROUP_ACS_2022_POP_COLUMN) + + block_weights_df = read_csv_s3_or_local( + paths.census_block_weights_path, + usecols=usecols, + dtype={BLOCK_GROUP_GEOID_COLUMN: 'string'}, + ) + + # Require the explicit state column to exist + if 'state_abb' not in block_weights_df.columns: + raise RuntimeError( + f"Census block weights CSV missing required column 'state_abb': {paths.census_block_weights_path}" + ) + + # Filter to rows for this state and fail if none found + state_block_weights = block_weights_df.loc[block_weights_df['state_abb'] == state_config.postal] + if state_block_weights.empty: + raise RuntimeError( + f'No census block weights found for state {state_config.postal} in {paths.census_block_weights_path}' + ) + + # Prepare block-group population according to version + if cfg.version_decimal >= Decimal('1.0'): + block_group_population = prepare_block_group_population_v1(state_block_weights) + else: + block_group_population = prepare_block_group_population_v0(state_block_weights) + + final_scores = build_final_scores(tract_scores, block_group_population, version=cfg.version_decimal) zero_population_count = int(block_group_population[BLOCK_GROUP_POP_COLUMN].eq(0).sum()) logging.info( 'Writing final Ozone scores to %s (rows=%d, zero_population_groups=%d)', @@ -434,21 +558,61 @@ def main(argv=None) -> int: """Run the Ozone tract-to-block-group indicator workflow.""" log_path = configure_logging() cfg = get_config(argv) + + # Parse and validate version early; fail fast for versions we don't support yet. + version_decimal = parse_version_decimal(cfg.version) + cfg = replace(cfg, version_decimal=version_decimal) + # Note that the following line should be updated every time the major version + # number is incremented. The design plan is that minor version increments (e.g. 1.0 -> 1.1) + # should happen when we have new data files but not new columns. + manifest_version_unsupported = Decimal('2.0') + if cfg.version_decimal is not None and cfg.version_decimal >= manifest_version_unsupported: + raise RuntimeError('Configured manifest version >= %s is not yet supported by this module' % manifest_version_unsupported) + + logging.info('Runtime config: version=%s storage_mode=%s state=%s output_dir=%s', + cfg.version_decimal, cfg.storage_mode, cfg.state or 'all', cfg.output_dir or 'None') initialize_runtime_dependencies(cfg) state_targets = load_state_targets(cfg.state) - tract_scores_path = join_root_and_relative_path( - get_active_o3_root_path(cfg.storage_mode), - get_o3_config().preprocessed_tract_output_relative_path, + # Load the score-stage manifest once and reuse it for all path resolution + manifest = build_manifest.get_stage_manifest( + target_type='indicator', + name='o3', + stage='score', + version=cfg.version, + environment=cfg.storage_mode, ) + tract_entry = manifest.get('inputs', {}).get('preprocessed_tracts') + if not tract_entry: + raise RuntimeError('Score manifest missing required input: preprocessed_tracts') + # Use the resolver to get an absolute indicator root (local) or raw remote root + indicator_root = resolve_path.get_indicator_root('o3', cfg.version, cfg.storage_mode) + tract_scores_path = join_root_and_relative_path(indicator_root, tract_entry['relative']) logging.info('Logging to %s', log_path) logging.info('Selected state filter: %s', cfg.state or 'all configured states') logging.info('Using tract scores path: %s', tract_scores_path) + + # If dry-run was requested, print resolved paths for the tract scores and + # for each state's weights and outputs, then exit without reading/writing. + if cfg.dry_run: + print('DRY RUN: manifest and path resolution') + print(f'tract_scores_path={tract_scores_path}') + logging.info('DRY RUN tract_scores_path=%s', tract_scores_path) + for state_config in state_targets: + paths = resolve_paths(cfg, state_config, manifest) + print(f'state={state_config.postal}') + print(f' census_block_weights_path={paths.census_block_weights_path}') + print(f' final_bg_scores_path={paths.final_bg_scores_path}') + logging.info('DRY RUN state=%s', state_config.postal) + logging.info('DRY RUN census_block_weights_path=%s', paths.census_block_weights_path) + logging.info('DRY RUN final_bg_scores_path=%s', paths.final_bg_scores_path) + return 0 tract_scores_df = read_csv_s3_or_local(tract_scores_path, dtype={TRACT_GEOID_COLUMN: 'string'}) prepared_tract_scores = prepare_tract_scores(tract_scores_df) logging.info('Prepared %d tract-level Ozone scores', len(prepared_tract_scores)) for state_config in state_targets: - process_state(cfg, state_config, prepared_tract_scores) + # pass manifest into process_state via resolve_paths to avoid duplicate lookups + process_state(cfg, state_config, prepared_tract_scores, manifest) state_count = len(state_targets) msg = f"Completed Ozone block-group indicator generation" diff --git a/scripts/o3/readme.md b/scripts/o3/readme.md index 20ba718..895a5c9 100644 --- a/scripts/o3/readme.md +++ b/scripts/o3/readme.md @@ -15,7 +15,7 @@ This folder contains the Ozone (O3) indicator pipeline and its state-validation ``` 2. `o3_preprocess.py` reads the compressed source directly and writes one annual average concentration per tract. -3. `o3_indicator.py` expands tract scores to block groups, applies the zero-population null rule, and writes per-state `final_bg_scores.csv` outputs. +3. `o3_score.py` expands tract scores to block groups, applies the zero-population null rule, and writes per-state `final_bg_scores.csv` outputs. 4. `run_state_validation.sh` reruns one state and, in local mode, compares the result to an Ozone-oriented EJAM subset file. ## Key files @@ -24,7 +24,7 @@ This folder contains the Ozone (O3) indicator pipeline and its state-validation - `o3_config.json`: O3 local and remote roots plus raw-download settings. - `o3_config.py`: validated config loader shared by the O3 scripts. - `o3_preprocess.py`: tract-level annual averaging step. -- `o3_indicator.py`: tract-to-block-group expansion and final state output step. +- `o3_score.py`: tract-to-block-group expansion and final state output step. - `run_state_validation.sh`: one-state validation wrapper. ## Current local validation flow diff --git a/scripts/pm25/pm25_config.json b/scripts/pm25/pm25_config.json index 0421b62..836525c 100644 --- a/scripts/pm25/pm25_config.json +++ b/scripts/pm25/pm25_config.json @@ -1,14 +1,67 @@ { - "local_root_path": "./pipeline/", - "remote_root_path": "s3://pedp-data-preserved/ejscreen-data-processing/pm25/pipeline/", - "preprocessed_tract_output_relative_path": "preprocessed_input/pm25_tract_annual_average.csv", - "indicator_output_relative_path_template": "output/indicators/{postal}", - "downloads": { + "local_root_path": "./pipeline/pm25", + "remote_root_path": "s3://pedp-data-preserved/ejscreen-data-processing/pipeline/pm25/", + "download_settings": { "request_timeout_seconds": 120, - "chunk_size_bytes": 1048576, - "raw_pm25_2020": { - "relative_path": "downloads/2020/2020_pm25_daily_average.txt.gz", - "source_url": "https://ofmpub.epa.gov/rsig/rsigserver?data/FAQSD/outputs/2020_pm25_daily_average.txt.gz" + "chunk_size_bytes": 1048576 + }, + "versions": { + "1.0": { + "required_shared_assets": { + "census_block_weights": "0.5" + }, + "stages": { + "fetch": { + "outputs": { + "raw_pm25": { + "type": "download", + "relative_path": "v1.0/downloads/2020/2020_pm25_daily_average.txt.gz", + "source_url": "https://ofmpub.epa.gov/rsig/rsigserver?data/FAQSD/outputs/2020_pm25_daily_average.txt.gz" + } + } + }, + "preprocess": { + "inputs": { + "primary_pm25": { + "type": "indicator_download", + "key": "raw_pm25" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v1.0/preprocessed_input/" + }, + "main_tract_averages": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/pm25_tract_annual_average.csv" + } + } + }, + "score": { + "inputs": { + "preprocessed_tracts": { + "type": "file", + "relative_path": "v1.0/preprocessed_input/pm25_tract_annual_average.csv" + }, + "weights": { + "type": "shared_asset", + "key": "census_block_weights", + "category": "preprocessed_input" + } + }, + "outputs": { + "output_directory": { + "type": "directory", + "relative_path": "v1.0/output/indicators/" + }, + "indicator_output_template": { + "type": "template", + "relative_path": "v1.0/output/indicators/{postal}/" + } + } + } + } } } } \ No newline at end of file diff --git a/scripts/pm25/pm25_migration_todo.md b/scripts/pm25/pm25_migration_todo.md new file mode 100644 index 0000000..f5b009c --- /dev/null +++ b/scripts/pm25/pm25_migration_todo.md @@ -0,0 +1,82 @@ +# PM2.5 Manifest Migration — Walkthrough TODO + +Created: 2026-06-10 +Purpose: Use the `o3` manifest-driven changes as a template to convert `pm25` to manifest/stage-based processing. +Composed and written in the voice of GPT-5 chatbot. Some editing done by AG. + +--- + +## Checklist + +- [ ] Diff `o3` between `indicators` and `dataVersioning` branches and develop a change template/checklist + - Status: not-started + - Goal: capture the exact changes made to `scripts/o3` (dataVersioning vs indicators) so we can port the pattern. + - Command (local): + + ```bash + git diff --no-color indicators..dataVersioning -- scripts/o3 + ``` + + - Output: short patch/notes to use as a coding template. + +- [ ] Inspect `scripts/pm25/pm25_config.json` and map further required changes + - Status: partially implemented + - Done: config matches the o3 architecture as of o3 initial hand-off (6/13). May need to be updated if changes to o3 config are made during testing. + +- [ ] Refactor `scripts/pm25/pm25_preprocess.py` to manifest-driven CLI + - Status: not-started + - Implementation notes: + - Replace direct `pm25_config` usage with `build_manifest.get_stage_manifest(target_type='indicator', name='pm25', stage='preprocess', version=..., environment=...)`. + - Use `resolve_path.get_indicator_root('pm25', version, storage_mode)` for roots. + - CLI: positional `storage_mode` (`local|remote`), `-v/--version` (default `1.0`), long-only `--dry-run`. + - Early exit on dry-run after validating manifest + headers. + - Files to edit: `scripts/pm25/pm25_preprocess.py`. + +- [ ] Refactor `scripts/pm25/pm25_indicator.py` to manifest-driven CLI + - Status: not-started + - Implementation notes: + - Add `-l/--location` required, `--state` required (accept `all` mapped to `None`), `-v/--version` default `1.0`, `--dry-run` long-only. + - Load the `score` stage manifest once in `main()` and pass `manifest` into `resolve_paths()`/`process_state()` to avoid duplicate lookups. + - Use `resolve_path.get_indicator_root('pm25', version, storage_mode)` and `resolve_path.get_dependency_version('pm25', version, 'census_block_weights')` + `resolve_path.get_shared_root(...)` to resolve shared inputs. + - Remove dependency on `scripts/pm25/pm25_config.py`; ensure `scripts/shared` importability via `sys.path` insertion as in `o3`. + - Files to edit: `scripts/pm25/pm25_indicator.py` (plus small helper imports). + +- [ ] Delete legacy `scripts/pm25/pm25_config.py` (once all references removed) + - Status: not-started + - Steps: + - Search for `pm25_config` imports across repo. + - Delete file and run syntax checks. + +- [ ] Run local tests and dry-run smoke tests + - Status: not-started + - Commands: + + ```bash + python -m py_compile scripts/pm25/pm25_preprocess.py scripts/pm25/pm25_indicator.py + python scripts/pm25/pm25_preprocess.py local --dry-run + python scripts/pm25/pm25_indicator.py --location local --state WY --dry-run + ``` + + - Check logs in `scripts/pm25/*.log` for resolved paths and dry-run messages. + +- [ ] Branching / PR + - Status: not-started + - Suggested branch: `feature/pm25-manifest-migration` + - Workflow: commit each logical change (config → preprocess → indicator → cleanup), run tests, push, open PR for review. + +--- + +## How we'll walk through this file together + +- I will update the checkboxes and add brief commit notes after each change. +- Please mark a step "go" by replying with the step name (e.g., `diff`, `preprocess`, `indicator`) and I will implement that step and run the local tests. +- If you prefer to implement interactively together, tell me which step to start and I will proceed and push small, testable commits. + +--- + +If you want this file moved to a different path (e.g., `docs/`), tell me and I will relocate it. If you'd like me to start with `diff` now, say `go: diff` and I will run the local diff and produce the template notes/patch. + +Last updated: 2026-06-13 + +Notes: +- Consolidated with the repo-root TODO; this file is now the single authoritative PM2.5 migration TODO. diff --git a/scripts/pm25/pm25_preprocess.py b/scripts/pm25/pm25_preprocess.py index ab68927..d4b934c 100644 --- a/scripts/pm25/pm25_preprocess.py +++ b/scripts/pm25/pm25_preprocess.py @@ -6,7 +6,7 @@ the tract-level intermediate CSV used by the indicator step. Process summary: - - Resolve local or remote PM2.5 storage from pm25_config.json. + - Resolve local or remote PM2.5 storage from the manifest + resolver. - Validate the compressed file header before performing the full read. - Read only the required columns from the raw source. - Validate tract GEOIDs, dates, and numeric PM2.5 values. @@ -16,6 +16,10 @@ Runtime arguments: - storage_mode Required. Either local or remote. + - --version / -v + Optional. Config version to use (default: 1.0). + - --dry-run + Optional. Validate manifest and headers, then exit without processing or writing outputs. Outputs: - preprocessed_input/pm25_tract_annual_average.csv under the active PM2.5 root. @@ -30,10 +34,19 @@ import importlib import logging from pathlib import Path +import sys import pandas as pd -from pm25_config import get_pm25_config, resolve_local_pm25_root_path +# Ensure the `scripts/shared` folder is importable so modules in that +# folder (e.g. build_manifest.py, resolve_path.py) can be imported as +# top-level modules when running this script directly. +SHARED_SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "shared" +if str(SHARED_SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SHARED_SCRIPTS_DIR)) + +import build_manifest +import resolve_path PM25_DIR = Path(__file__).resolve().parent @@ -44,19 +57,28 @@ TRACT_GEOID_COLUMN = 'tract_geoid' ANNUAL_AVERAGE_COLUMN = 'annual_average_concentration' +# Centralize frequently used regex/pattern constants. +FIPS_PATTERN = r'\d{11}' + @dataclass(frozen=True, slots=True) class Config: storage_mode: str + version: str local_root_path: str remote_root_path: str raw_download_relative_path: str preprocessed_tract_output_relative_path: str + dry_run: bool = False def get_config(argv=None) -> Config: - """Parse runtime arguments and merge them with validated PM2.5 config values.""" - raw_config = get_pm25_config() + """Parse runtime arguments and derive canonical paths from the manifest/resolver. + + This function uses `build_manifest.get_stage_manifest()` (preprocess stage) + and `resolve_path.get_indicator_root()` to populate the runtime config. + No compatibility fallback to the old flat JSON format is provided. + """ parser = argparse.ArgumentParser( description='Read the raw national PM2.5 .txt.gz file, compute tract-level annual average concentration, and write the preprocessed tract CSV.' @@ -66,14 +88,60 @@ def get_config(argv=None) -> Config: choices=('local', 'remote'), help='Select whether the script reads and writes through the configured local root path or remote S3 root path.', ) + parser.add_argument( + '-v', '--version', + dest='version', + default='1.0', + help='Optional: config version to base processing on (current default: 1.0)' + ) + # Long-only dry-run flag (no short alias) + parser.add_argument( + '--dry-run', + dest='dry_run', + action='store_true', + help='Dry run: validate manifest and headers but do not process or write outputs.', + ) args = parser.parse_args(argv) + manifest = build_manifest.get_stage_manifest( + target_type='indicator', + name='pm25', + stage='preprocess', + version=args.version, + environment=args.storage_mode, + ) + + inputs = manifest.get('inputs', {}) + outputs = manifest.get('outputs', {}) + + if 'primary_pm25' not in inputs: + raise RuntimeError('Preprocess manifest missing required input: primary_pm25') + if 'main_tract_averages' not in outputs: + raise RuntimeError('Preprocess manifest missing required output: main_tract_averages') + + # Manifest entries may already include a compiled "root" and "relative". + raw_entry = inputs['primary_pm25'] + preproc_entry = outputs['main_tract_averages'] + + raw_rel = raw_entry.get('relative') + preproc_rel = preproc_entry.get('relative') + + if not raw_rel or not isinstance(raw_rel, str): + raise RuntimeError('Invalid relative path for primary_pm25 in preprocess manifest') + if not preproc_rel or not isinstance(preproc_rel, str): + raise RuntimeError('Invalid relative path for main_tract_averages in preprocess manifest') + + local_root = resolve_path.get_indicator_root('pm25', args.version, 'local') + remote_root = resolve_path.get_indicator_root('pm25', args.version, 'remote') + return Config( storage_mode=args.storage_mode, - local_root_path=resolve_local_pm25_root_path(PM25_DIR), - remote_root_path=raw_config.remote_root_path, - raw_download_relative_path=raw_config.raw_download_relative_path, - preprocessed_tract_output_relative_path=raw_config.preprocessed_tract_output_relative_path, + version=args.version, + local_root_path=local_root, + remote_root_path=remote_root, + raw_download_relative_path=raw_rel, + preprocessed_tract_output_relative_path=preproc_rel, + dry_run=args.dry_run, ) @@ -198,7 +266,7 @@ def validate_and_prepare_raw_table(df: pd.DataFrame) -> pd.DataFrame: require_columns(df, (RAW_FIPS_COLUMN, RAW_DATE_COLUMN, RAW_PM25_COLUMN)) prepared = df[[RAW_FIPS_COLUMN, RAW_DATE_COLUMN, RAW_PM25_COLUMN]].copy() prepared[RAW_FIPS_COLUMN] = prepared[RAW_FIPS_COLUMN].astype('string').str.strip() - invalid_fips_mask = prepared[RAW_FIPS_COLUMN].isna() | ~prepared[RAW_FIPS_COLUMN].str.fullmatch(r'\d{11}') + invalid_fips_mask = prepared[RAW_FIPS_COLUMN].isna() | ~prepared[RAW_FIPS_COLUMN].str.fullmatch(FIPS_PATTERN) if invalid_fips_mask.any(): invalid_samples = prepared.loc[invalid_fips_mask, RAW_FIPS_COLUMN].drop_duplicates().astype(str).head(5).tolist() raise RuntimeError( @@ -271,6 +339,18 @@ def main(argv=None) -> int: header_df = read_raw_pm25_header(raw_input_path) require_columns(header_df, (RAW_FIPS_COLUMN, RAW_DATE_COLUMN, RAW_PM25_COLUMN)) logging.info('Raw PM2.5 header validation passed') + # If dry-run was requested, we've validated manifest resolution and the + # raw file header; exit early without performing the full read/aggregation + # or writing outputs. + if cfg.dry_run: + logging.info('Dry run enabled; skipping full read/processing and write.') + logging.info('DRY RUN: manifest and header validated.') + logging.info('raw_input_path=%s', raw_input_path) + logging.info('output_path=%s', output_path) + print('DRY RUN: manifest and header validated.') + print(f'raw_input_path={raw_input_path}') + print(f'output_path={output_path}') + return 0 logging.info('Reading raw PM2.5 input table') raw_df = read_raw_pm25_table(raw_input_path) @@ -283,4 +363,4 @@ def main(argv=None) -> int: if __name__ == '__main__': - raise SystemExit(main()) \ No newline at end of file + raise SystemExit(main()) diff --git a/scripts/resolve_path_specification.md b/scripts/resolve_path_specification.md new file mode 100644 index 0000000..e0ad5f7 --- /dev/null +++ b/scripts/resolve_path_specification.md @@ -0,0 +1,131 @@ +# Architectural Specifications: Path Resolution System + +This document contains three distinct sections outlining the design, validation, and integration of the centralized file path resolution engine for the data processing pipeline. + +--- + +## Section 1: Module Specification for `resolve_path.py` +**Target Audience:** Coding AI / Code Generator +**Objective:** Implement a centralized configuration-parsing and path-resolution module. The focus is on the interface boundaries, performance constraints, and functional requirements. Do not prescribe the specific internal dictionary-traversal or string-building algorithms. + +### 1. Architectural & Interoperability Requirements +* **Language Environment:** Pure Python 3.x using only standard libraries (`json`, `os`, `pathlib`). No external library dependencies. +* **R Compatibility:** The module must support execution from R via the `reticulate` package. It must expose a flat functional interface at the module root level. Under the hood, these functions should map to a single, persistent internal class instance (Singleton pattern) to cache file-system I/O. +* **Directory Context:** The module will always be executed from the `project_root/` folder context. +* **Module Location:** The resolver module will live at `./scripts/shared/resolve_path.py`. + +### 2. Output Data Shape (The Dictionary Return Contract) +All functions that resolve asset storage locations must return a **Python Dictionary** that includes at least the following keys: +```python +{ + "root": "string representing the base configuration path", + "relative": "string representing the version-specific relative path" +} + +``` + +* **Local Context:** When `environment="local"`, the `"root"` value must be extracted from the configuration's top-level `local_root_path`. +* **Remote Context:** When `environment="remote"`, the `"root"` value must be extracted from the configuration's top-level `remote_root_path`. +* **Environment Validation:** `environment` must accept only `"local"` or `"remote"`. Any other value must raise an explicit error immediately. +* **String Safety:** S3 paths (remote) must not be processed using standard Windows-style path utilities (`os.path.join`) to avoid corrupting forward slashes into backslashes. Use cloud-safe string or posix path formatting for remote resolution. + +### 3. Target Configuration Files + +The module must parse configurations from two known locations based on the input arguments: + +1. **Indicator Configuration:** Located at `./scripts/{indicator}/{indicator}_config.json` +2. **Shared Asset Configuration:** Located at `./scripts/shared/shared_config.json` + +### 4. Required Module Interface (Exposed Functions) + +#### `get_download_path(indicator: str, version: str, asset_key: str, environment: str = "local") -> dict` + +* **What it does:** Looks up an indicator-specific fetch-stage output asset. +* **Behavior:** Traverses the indicator's configuration file for the matching `version`, navigates to `stages -> fetch -> outputs`, and extracts the target `asset_key` information. +* **Return Value:** A dictionary that includes `"root"` and `"relative"` keys mapped to the selected environment settings. + +#### `get_dependency_version(indicator: str, version: str, dependency: str) -> str` + +* **What it does:** Looks up which version of a shared asset an indicator requires. +* **Behavior:** Traverses the indicator's configuration file for the specified `version` and extracts the precise version string pinned under the `required_shared_assets` mapping for that `dependency`. +* **Return Value:** A plain string representing the version number (e.g., `"1.0"`, `"2022.1"`). + +#### `get_shared_asset_path(asset: str, version: str, category: str, asset_key: str = None, environment: str = "local") -> dict` + +* **What it does:** Looks up paths inside the global `shared_config.json`. +* **Behavior:** Targets the specific `asset` and its nested `version`. +* If `category` is `"preprocessed_input"`, it resolves the location from `stages -> preprocess -> outputs` using the asset output's `relative_path_template`. +* If `category` is `"downloads"`, it requires the `asset_key` and resolves from `stages -> fetch -> outputs` using the selected output's `relative_path_template`. + + +* **Return Value:** A dictionary that includes `"root"` and `"relative"` keys. +* *Note: Any wildcard templates containing string tokens like `{postal}` or `{fips}` must be returned intact within the `"relative"` string; the resolver does not perform token interpolation.* + +--- + +## Section 2: Command-Line Test Harness (`test_resolve_path.py`) + +**Target Audience:** System Validator / Integration Tester +**Objective:** Provide a command-line script that executes a "dry-run" configuration check against the path resolver. It prints out parsed results so that engineers can verify paths instantly without spinning up a full processing run. + +### 1. Interface Requirements + +The harness must be a standalone executable Python script located at `./scripts/test_harness/test_resolve_path.py` that implements Python’s standard `argparse` library. It must accept **4 core named arguments**, alongside contextual switches for sub-keys. + +### 2. Command-Line Arguments + +* `--storage-mode` (Required): String matching either `local` or `remote`. +* `--type` (Required): String matching either `indicator` or `shared`. Determines which underlying resolver mechanics to trigger. +* `--name` (Required): String name of the asset domain (e.g., `o3`, `pm25`, `tiger_bg`, `census_block_weights`). +* `--version` (Required): String version identifier (e.g., `1.0`, `2020.1`). +* `--key` (Optional): String key name representing an item's sub-asset key (e.g., `raw_o3` for indicators, or `raw_weight_crosswalks` for shared downloads). +* `--category` (Optional): String matching either `downloads` or `preprocessed_input` (used strictly when `--type` is `shared`). +* **Validation Rule:** The harness should validate command shape, but invalid `--storage-mode` and invalid shared `--category` values must be passed through so the resolver module itself performs the fast-fail semantic validation. + +### 3. Execution Behavior & Sample Output + +The script must import `resolve_path.py` from `./scripts/shared`, determine the user's intent from the switches, call the appropriate resolver function, and print the resulting dictionary to standard output as a formatted JSON block. + +The optional `"status"` field shown in the examples below belongs to the harness output, not to the resolver contract itself. + +#### Example Command 1 (Indicator Verification): + +```bash +python ./scripts/test_harness/test_resolve_path.py --storage-mode remote --type indicator --name o3 --version 1.0 --key raw_o3 + +``` + +#### Expected Output stdout: + +```json +{ + "status": "DRY_RUN_RESOLVED", + "root": "s3://pedp-data-preserved/ejscreen-data-processing/pipeline/o3/", + "relative": "v1.0/downloads/2020/2020_ozone_daily_8hour_maximum.txt.gz" +} + +``` + +#### Example Command 2 (Shared Preprocessed Asset Verification): + +```bash +python ./scripts/test_harness/test_resolve_path.py --storage-mode local --type shared --name census_block_weights --version 1.0 --category preprocessed_input + +``` + +#### Expected Output stdout: + +```json +{ + "status": "DRY_RUN_RESOLVED", + "root": "./pipeline/shared/", + "relative": "census_block_weights/1.0/preprocessed_input/census_block_weights_2020_{postal}.csv" +} + +``` + +--- + +--- + +**Note:** Section 3 (R Developer Integration Guide) has been moved to a dedicated document to simplify integration references for R users: see `scripts/R_programmer_guide.md`. diff --git a/scripts/shared/build_manifest.py b/scripts/shared/build_manifest.py new file mode 100644 index 0000000..9d1cb43 --- /dev/null +++ b/scripts/shared/build_manifest.py @@ -0,0 +1,343 @@ +"""Strict stage manifest builder for indicator and shared workflows. + +Public API: +- get_stage_manifest(target_type: str, name: str, stage: str, version: str, environment: str = "local") -> dict + +Behavior: +- Compiles a stage-level manifest for an `indicator` or `shared` target and + returns a dictionary with `inputs` and `outputs`. Each entry is compiled to + include at minimum `root` and `relative` values; fetch entries may include + additional metadata such as `source_url`, `scope`, or `source_url_template`. +- When `environment == 'local'` compiled `root` values are resolved against the + project root. `environment` accepts only 'local' or 'remote'. + +Notes: +- The builder delegates shared-version validation to `resolve_path.get_shared_version_block` + to ensure one authoritative implementation for shared-asset lookups and error messages. +- Exposes a flat function `get_stage_manifest` suitable for use from R via `reticulate`. +""" + +from __future__ import annotations + +import json +import logging +from pathlib import Path + +import resolve_path + + +LOGGER = logging.getLogger(__name__) +LOGGER.addHandler(logging.NullHandler()) + +SCRIPTS_ROOT = Path(__file__).resolve().parent.parent +VALID_ENVIRONMENTS = {"local", "remote"} +VALID_TARGET_TYPES = {"indicator", "shared"} +SHARED_CONFIG_PATH = Path(__file__).with_name("shared_config.json") + + +class _ManifestBuilder: + def __init__(self) -> None: + self._indicator_config_cache: dict[str, dict[str, object]] = {} + self._shared_config_cache: dict[str, object] | None = None + + def get_stage_manifest( + self, + target_type: str, + name: str, + stage: str, + version: str, + environment: str = "local", + ) -> dict[str, dict[str, dict[str, str]]]: + self._validate_target_type(target_type) + self._validate_environment(environment) + + if target_type == "indicator": + config = self._load_indicator_config(name) + version_block = self._get_indicator_version_block(config, name, version) + stage_block = self._get_indicator_stage_block(version_block, name, version, stage) + return { + "inputs": self._compile_indicator_inputs(config, stage_block, name, version, environment), + "outputs": self._compile_target_outputs( + config, + stage_block, + f"{name}.versions.{version}.stages.{stage}", + environment, + ), + } + + config = self._load_shared_config() + version_block = self._get_shared_version_block(config, name, version) + stage_block = self._get_shared_stage_block(version_block, name, version, stage) + manifest = { + "inputs": self._compile_shared_inputs(stage_block, name, version), + "outputs": self._compile_target_outputs( + config, + stage_block, + f"shared.assets.{name}.{version}.stages.{stage}", + environment, + ), + } + return manifest + + def _compile_indicator_inputs( + self, + config: dict[str, object], + stage_block: dict[str, object], + indicator: str, + version: str, + environment: str, + ) -> dict[str, dict[str, str]]: + inputs = self._optional_mapping(stage_block.get("inputs"), "inputs") + compiled: dict[str, dict[str, str]] = {} + + for input_name, entry in inputs.items(): + field_prefix = f"{indicator}.versions.{version}.stages.*.inputs.{input_name}" + entry_mapping = self._require_mapping(entry, field_prefix) + entry_type = self._require_non_empty_string(entry_mapping.get("type"), f"{field_prefix}.type") + + if entry_type == "indicator_download": + asset_key = self._require_non_empty_string(entry_mapping.get("key"), f"{field_prefix}.key") + compiled[input_name] = resolve_path.get_download_path(indicator, version, asset_key, environment) + continue + + if entry_type == "shared_asset": + asset_name = self._require_non_empty_string(entry_mapping.get("key"), f"{field_prefix}.key") + category = self._require_non_empty_string(entry_mapping.get("category"), f"{field_prefix}.category") + shared_version = resolve_path.get_dependency_version(indicator, version, asset_name) + shared_asset_key = None + if category == "downloads": + shared_asset_key = self._require_non_empty_string( + entry_mapping.get("asset_key"), + f"{field_prefix}.asset_key", + ) + compiled[input_name] = resolve_path.get_shared_asset_path( + asset=asset_name, + version=shared_version, + category=category, + asset_key=shared_asset_key, + environment=environment, + ) + continue + + if entry_type in {"file", "directory", "template"}: + compiled[input_name] = self._compile_indicator_relative_entry( + config, + entry_mapping, + field_prefix, + environment, + ) + continue + + self._fail(f"Unsupported input type {entry_type!r} at {field_prefix}.type") + + return compiled + + def _compile_indicator_relative_entry( + self, + config: dict[str, object], + entry_mapping: dict[str, object], + field_prefix: str, + environment: str, + ) -> dict[str, str]: + # Indicator inputs that reference local files/directories/templates + # reuse the same relative entry compilation used for outputs. + return self._compile_relative_entry(config, entry_mapping, field_prefix, environment) + + def _compile_shared_inputs( + self, + stage_block: dict[str, object], + asset_name: str, + version: str, + ) -> dict[str, dict[str, str]]: + inputs = self._optional_mapping(stage_block.get("inputs"), "inputs") + if not inputs: + return {} + + for input_name in inputs: + self._fail( + f"Shared target inputs are not yet defined for strict manifest compilation: " + f"shared.assets.{asset_name}.{version}.stages.*.inputs.{input_name}" + ) + + return {} + + def _compile_target_outputs( + self, + config: dict[str, object], + stage_block: dict[str, object], + field_prefix: str, + environment: str, + ) -> dict[str, dict[str, str]]: + outputs = self._optional_mapping(stage_block.get("outputs"), "outputs") + compiled: dict[str, dict[str, str]] = {} + + for output_name, entry in outputs.items(): + output_prefix = f"{field_prefix}.outputs.{output_name}" + entry_mapping = self._require_mapping(entry, field_prefix) + compiled[output_name] = self._compile_relative_entry( + config, + entry_mapping, + output_prefix, + environment, + ) + + return compiled + + def _compile_relative_entry( + self, + config: dict[str, object], + entry_mapping: dict[str, object], + field_prefix: str, + environment: str, + ) -> dict[str, str]: + root = self._get_root(config, environment, "target") + relative = self._extract_relative_value(entry_mapping, field_prefix) + compiled: dict[str, str] = {"root": root, "relative": relative} + + # Optionally copy fetch/source metadata when present (fetch outputs + # may define `source_url`, `source_url_template`, and `scope`). + source_url = entry_mapping.get("source_url") + if isinstance(source_url, str) and source_url: + compiled["source_url"] = source_url + source_template = entry_mapping.get("source_url_template") + if isinstance(source_template, str) and source_template: + compiled["source_url_template"] = source_template + scope = entry_mapping.get("scope") + if isinstance(scope, str) and scope: + compiled["scope"] = scope + + return compiled + + def _extract_relative_value(self, entry_mapping: dict[str, object], field_prefix: str) -> str: + relative_path = entry_mapping.get("relative_path") + relative_template = entry_mapping.get("relative_path_template") + + if isinstance(relative_path, str) and relative_path: + return relative_path + if isinstance(relative_template, str) and relative_template: + return relative_template + + self._fail( + f"Expected {field_prefix} to define a non-empty relative_path or relative_path_template" + ) + + def _load_indicator_config(self, indicator: str) -> dict[str, object]: + if indicator in self._indicator_config_cache: + return self._indicator_config_cache[indicator] + + config_path = SCRIPTS_ROOT / indicator / f"{indicator}_config.json" + config = self._load_json_object(config_path, f"indicator {indicator}") + self._indicator_config_cache[indicator] = config + return config + + def _load_shared_config(self) -> dict[str, object]: + if self._shared_config_cache is not None: + return self._shared_config_cache + + self._shared_config_cache = self._load_json_object(SHARED_CONFIG_PATH, "shared") + return self._shared_config_cache + + def _load_json_object(self, config_path: Path, label: str) -> dict[str, object]: + if not config_path.exists(): + self._fail(f"Missing {label} config file: {config_path}") + + LOGGER.info("Loading %s config from %s", label, config_path) + with config_path.open("r", encoding="utf-8") as handle: + payload = json.load(handle) + + if not isinstance(payload, dict): + self._fail(f"Expected {label} config to be a JSON object: {config_path}") + return payload + + def _get_indicator_version_block( + self, + config: dict[str, object], + indicator: str, + version: str, + ) -> dict[str, object]: + versions = self._require_mapping(config.get("versions"), f"{indicator}.versions") + version_block = versions.get(version) + if version_block is None: + self._fail(f"Version {version!r} not found for indicator {indicator}") + return self._require_mapping(version_block, f"{indicator}.versions.{version}") + + def _get_indicator_stage_block( + self, + version_block: dict[str, object], + indicator: str, + version: str, + stage: str, + ) -> dict[str, object]: + stages = self._require_mapping(version_block.get("stages"), f"{indicator}.versions.{version}.stages") + return self._require_mapping(stages.get(stage), f"{indicator}.versions.{version}.stages.{stage}") + + def _get_shared_version_block( + self, + config: dict[str, object], + asset_name: str, + version: str, + ) -> dict[str, object]: + # Delegate shared-version validation to the centralized resolver + # to avoid duplicating lookup logic and error messages. + return resolve_path.get_shared_version_block(config, asset_name, version) + + def _get_shared_stage_block( + self, + version_block: dict[str, object], + asset_name: str, + version: str, + stage: str, + ) -> dict[str, object]: + stages = self._require_mapping(version_block.get("stages"), f"shared.assets.{asset_name}.{version}.stages") + return self._require_mapping(stages.get(stage), f"shared.assets.{asset_name}.{version}.stages.{stage}") + + def _get_root(self, config: dict[str, object], environment: str, label: str) -> str: + field_name = "local_root_path" if environment == "local" else "remote_root_path" + return self._require_non_empty_string(config.get(field_name), f"{label}.{field_name}") + + def _validate_environment(self, environment: str) -> None: + if environment not in VALID_ENVIRONMENTS: + self._fail(f"Invalid environment {environment!r}; expected one of {sorted(VALID_ENVIRONMENTS)}") + + def _validate_target_type(self, target_type: str) -> None: + if target_type not in VALID_TARGET_TYPES: + self._fail(f"Invalid target_type {target_type!r}; expected one of {sorted(VALID_TARGET_TYPES)}") + + def _optional_mapping(self, value: object, field_name: str) -> dict[str, object]: + if value is None: + return {} + return self._require_mapping(value, field_name) + + def _require_mapping(self, value: object, field_name: str) -> dict[str, object]: + if not isinstance(value, dict): + self._fail(f"Expected {field_name} to be a JSON object") + return value + + def _require_non_empty_string(self, value: object, field_name: str) -> str: + if not isinstance(value, str) or not value: + self._fail(f"Expected {field_name} to be a non-empty string") + return value + + def _fail(self, message: str) -> None: + LOGGER.error(message) + raise ValueError(message) + + +_BUILDER: _ManifestBuilder | None = None + + +def _get_builder() -> _ManifestBuilder: + global _BUILDER + if _BUILDER is None: + _BUILDER = _ManifestBuilder() + return _BUILDER + + +def get_stage_manifest( + target_type: str, + name: str, + stage: str, + version: str, + environment: str = "local", +) -> dict[str, dict[str, dict[str, str]]]: + return _get_builder().get_stage_manifest(target_type, name, stage, version, environment) \ No newline at end of file diff --git a/scripts/shared/fetch_raw.py b/scripts/shared/fetch_raw.py index 1e0acca..1df42e7 100644 --- a/scripts/shared/fetch_raw.py +++ b/scripts/shared/fetch_raw.py @@ -2,26 +2,25 @@ Purpose: Central fetch utility to download raw source files needed to generate indicator - scores. It depends on the indicator folder containing a config .json file with - the necessary `downloads` structure to specify the source and destination for - the download. + scores. It depends on the stage-based indicator and shared config .json files + to specify fetch outputs, source locations, and destination paths. Usage examples: # Simple indicator with a default download - python scripts/fetch_raw.py --indicator pm25 --location local + python shared/fetch_raw.py --indicator pm25 --storage-mode local # Remote fetch for ozone - python scripts/fetch_raw.py --indicator o3 --location remote + python shared/fetch_raw.py --indicator o3 --storage-mode remote # Shared tiger BG download for Vermont (postal code). --state uses two-letter postal codes. - python scripts/fetch_raw.py --indicator shared --download tiger_bg_2020 --state VT --location local + python shared/fetch_raw.py --indicator shared --download tiger_bg_2020 --state VT --storage-mode local Notes: - - `--location`/`-l` is required and must be either `local` or `remote`. - - `--download` must be specified when the indicator config includes multiple download entries. - See the pm25_config.json for an example of a single entry and shared_config.json for an - example set up for multiple entries. + - `--storage-mode`/`-l` is required and must be either `local` or `remote`. + - `--download` selects a fetch-stage output key. It is required when the selected target has + multiple fetch outputs, and for shared fetches. + - `--version` is optional only when the selected target has exactly one configured version. - `--state` - For state-scoped downloads use `--state` with a two-letter postal code which will be validated against `scripts/shared/state_config.json` and translated to the FIPS code @@ -39,6 +38,7 @@ from datetime import datetime import importlib.machinery import importlib.util +import json import sys import logging from pathlib import Path @@ -47,6 +47,9 @@ import requests +import build_manifest +import resolve_path + SCRIPTS_DIR = Path(__file__).resolve().parent PROCESS_NAME = 'fetch_raw' DEFAULT_LOG_FILENAME = 'fetch_raw.log' @@ -72,6 +75,7 @@ @dataclass(frozen=True, slots=True) class Config: indicator: str + version: str storage_mode: str local_root_path: str remote_root_path: str @@ -95,26 +99,6 @@ class DownloadResult: bytes_downloaded: int message: str -# This looks like a lot of code to simply load a module. -# But, since the module we want depends on our --indicator runtime -# argument, we need dynamic loading rather than static. -def _load_indicator_config_module(indicator: str): - # Indicator configs live at scripts//_config.py - cfg_root = SCRIPTS_DIR.parent - cfg_path = cfg_root / indicator / f'{indicator}_config.py' - if not cfg_path.exists(): - raise FileNotFoundError(f'Indicator config not found: {cfg_path}') - loader = importlib.machinery.SourceFileLoader(f'{indicator}_config', str(cfg_path)) - spec = importlib.util.spec_from_loader(loader.name, loader) - module = importlib.util.module_from_spec(spec) - # Ensure the module is present in sys.modules so decorators and runtime - # introspection (e.g. dataclasses) can locate the module object during - # execution. This mirrors importlib.import_module behavior. - sys.modules[loader.name] = module - loader.exec_module(module) - return module - - def _load_shared_state_module(): """Load the canonical shared state_config.py as a module and return it. @@ -133,52 +117,160 @@ def _load_shared_state_module(): return module +def _load_json_object(config_path: Path, label: str) -> dict[str, object]: + if not config_path.exists(): + raise FileNotFoundError(f'{label} config not found: {config_path}') + with config_path.open('r', encoding='utf-8') as handle: + payload = json.load(handle) + if not isinstance(payload, dict): + raise ValueError(f'{label} config must contain a JSON object: {config_path}') + return payload + + +def _require_mapping(value: object, field_name: str) -> dict[str, object]: + if not isinstance(value, dict): + raise ValueError(f'Expected {field_name} to be a JSON object') + return value + + +def _require_non_empty_string(value: object, field_name: str) -> str: + if not isinstance(value, str) or not value: + raise ValueError(f'Expected {field_name} to be a non-empty string') + return value + + +def _require_positive_int(value: object, field_name: str) -> int: + if not isinstance(value, int) or value <= 0: + raise ValueError(f'Expected {field_name} to be a positive integer') + return value + + +# local root resolution is now provided by `resolve_path.get_indicator_root` +# and `resolve_path.get_shared_root`. + + +def _resolve_version(versions: dict[str, object], label: str, requested_version: str | None) -> str: + if requested_version: + if requested_version not in versions: + raise ValueError(f'Unknown version {requested_version!r} for {label}') + return requested_version + + available_versions = sorted(versions.keys()) + if len(available_versions) != 1: + joined = ', '.join(available_versions) + raise ValueError(f'{label} requires an explicit --version. Available versions: {joined}') + return available_versions[0] + + +def _get_download_settings(config: dict[str, object], field_prefix: str) -> tuple[int, int]: + settings = _require_mapping(config.get('download_settings'), f'{field_prefix}.download_settings') + timeout = _require_positive_int(settings.get('request_timeout_seconds'), f'{field_prefix}.download_settings.request_timeout_seconds') + chunk_size = _require_positive_int(settings.get('chunk_size_bytes'), f'{field_prefix}.download_settings.chunk_size_bytes') + return timeout, chunk_size + + +def _resolve_fetch_output_key(outputs: dict[str, dict[str, str]], requested_key: str | None, label: str) -> str: + if requested_key: + if requested_key not in outputs: + available = ', '.join(sorted(outputs.keys())) + raise ValueError(f'Unknown fetch output key for {label}: {requested_key}. Available keys: {available}') + return requested_key + + if len(outputs) != 1: + available = ', '.join(sorted(outputs.keys())) + raise ValueError(f'{label} requires --download because multiple fetch outputs are available: {available}') + return next(iter(outputs)) + + +def _resolve_shared_fetch_target(shared_config: dict[str, object], download_key: str, requested_version: str | None) -> tuple[str, str]: + assets = _require_mapping(shared_config.get('assets'), 'shared.assets') + matches: list[tuple[str, str, dict[str, object]]] = [] + + for asset_name, versions_obj in assets.items(): + if not isinstance(versions_obj, dict): + continue + for version, version_block in versions_obj.items(): + if requested_version and version != requested_version: + continue + if not isinstance(version_block, dict): + continue + stages = version_block.get('stages') + if not isinstance(stages, dict): + continue + fetch_stage = stages.get('fetch') + if not isinstance(fetch_stage, dict): + continue + outputs = fetch_stage.get('outputs') + if not isinstance(outputs, dict): + continue + output_entry = outputs.get(download_key) + if isinstance(output_entry, dict): + matches.append((asset_name, version, output_entry)) + + if not matches: + if requested_version: + raise ValueError(f'Unknown shared fetch output key {download_key!r} for version {requested_version!r}') + raise ValueError(f'Unknown shared fetch output key {download_key!r}') + + if len(matches) > 1: + joined = ', '.join(f'{asset}@{version}' for asset, version, _ in matches) + raise ValueError( + f'Shared fetch output key {download_key!r} is ambiguous across targets: {joined}. Supply --version.' + ) + + # Return only the asset name and version; callers should look up the + # specific output entry from the manifest rather than relying on raw + # config objects. + asset_name, version, _ = matches[0] + return asset_name, version + + +def _resolve_fetch_source_and_relative( + entry: dict[str, object], + manifest_relative: str, + validated_postal: str | None, + validated_fips: str | None, + state_all_requested: bool, +) -> tuple[str, str, bool]: + source_url = entry.get('source_url') + source_url_template = entry.get('source_url_template') + is_state_scoped = bool(entry.get('scope') == 'state' or isinstance(source_url_template, str)) + + if is_state_scoped: + if not isinstance(source_url_template, str) or not source_url_template: + raise ValueError('State-scoped fetch output must define source_url_template') + if state_all_requested: + return manifest_relative, source_url_template, True + if not validated_postal or not validated_fips: + raise ValueError('This fetch output requires a valid --state/-s two-letter postal code') + try: + relative = manifest_relative.format(fips=validated_fips, postal=validated_postal) + source = source_url_template.format(fips=validated_fips, postal=validated_postal) + except Exception as exc: + raise ValueError(f'Failed to expand state templates for fetch output: {exc}') + return relative, source, False + + if not isinstance(source_url, str) or not source_url: + raise ValueError('Fetch output must define source_url') + return manifest_relative, source_url, False + + def get_config(argv=None) -> Config: parser = argparse.ArgumentParser(description='Centralized raw data fetch utility.') # Required arguments (listed first) parser.add_argument('-i', '--indicator', required=True, help='Required: Indicator key such as pm25 or shared') - parser.add_argument('-l', '--location', dest='storage_mode', choices=('local', 'remote'), required=True, help='Required: Select storage location (local or remote)') + parser.add_argument('-l', '--location', '--storage-mode', dest='storage_mode', choices=('local', 'remote'), required=True, help='Required: Select storage location (local or remote)') # Optional arguments - parser.add_argument('-d', '--download', help='Optional: Download key from the indicator config') + parser.add_argument('-d', '--download', help='Optional: Fetch-stage output key from the stage-based config') + parser.add_argument('-v', '--version', help='Optional: Explicit config version; required when multiple versions exist') parser.add_argument('-s', '--state', dest='state', help='Optional: Two-letter postal state code for state-scoped downloads (e.g. VT)') parser.add_argument('--dry-run', action='store_true', help='Optional: Print expanded source URL and destination path and exit') args = parser.parse_args(argv) indicator = args.indicator - cfg_module = _load_indicator_config_module(indicator) - - # Expect the indicator config to expose a get__config() helper and - # a resolve_local__root_path(pm_dir) helper similar to PM2.5. - get_cfg_fn_name = f'get_{indicator}_config' - resolve_local_fn_name = f'resolve_local_{indicator}_root_path' - if not hasattr(cfg_module, get_cfg_fn_name): - raise AttributeError(f'Indicator config module missing {get_cfg_fn_name}') - if not hasattr(cfg_module, resolve_local_fn_name): - raise AttributeError(f'Indicator config module missing {resolve_local_fn_name}') - - get_cfg_fn = getattr(cfg_module, get_cfg_fn_name) - resolve_local_fn = getattr(cfg_module, resolve_local_fn_name) - raw_config = get_cfg_fn() - - # Default values (used when --download not provided) - raw_download_relative_path = getattr(raw_config, 'raw_download_relative_path', None) - source_url = getattr(raw_config, 'source_url', None) - request_timeout_seconds = getattr(raw_config, 'request_timeout_seconds', 120) - chunk_size_bytes = getattr(raw_config, 'chunk_size_bytes', 1048576) - - # Determine the repo-level scripts root (runner lives in scripts/shared) - SCRIPTS_ROOT = SCRIPTS_DIR.parent - - # If a download key was provided, validate it against the indicator JSON - # configuration (this mirrors the pm25_config.json structure). - json_cfg_path = SCRIPTS_ROOT / indicator / f'{indicator}_config.json' - indicator_raw = None - if json_cfg_path.exists(): - import json - with json_cfg_path.open('r', encoding='utf-8') as fh: - indicator_raw = json.load(fh) + scripts_root = SCRIPTS_DIR.parent # If the user provided a --state value, validate it early and obtain the # canonical postal and fips values from the shared state config. Support @@ -187,7 +279,6 @@ def get_config(argv=None) -> Config: validated_postal = None validated_fips = None state_all_requested = False - state_all_mode = False if getattr(args, 'state', None): if isinstance(args.state, str) and args.state.strip().lower() == 'all': state_all_requested = True @@ -202,106 +293,74 @@ def get_config(argv=None) -> Config: except Exception as exc: raise ValueError(f'Invalid --state value {args.state!r}: {exc}') - if args.download: - if not json_cfg_path.exists(): - raise FileNotFoundError(f'Indicator JSON config not found for download validation: {json_cfg_path}') - import json - with json_cfg_path.open('r', encoding='utf-8') as fh: - indicator_raw = json.load(fh) - downloads_obj = indicator_raw.get('downloads') - if not isinstance(downloads_obj, dict) or not downloads_obj: - raise ValueError(f'{json_cfg_path.name} missing a non-empty "downloads" object') - # When the JSON includes top-level settings like request_timeout_seconds - # the actual entries live alongside them; collect entry keys. - # Accept either the flat layout used by pm25 (keys directly under - # downloads) or a nested `entries` object in future configs. - entries = {} - if 'entries' in downloads_obj and isinstance(downloads_obj['entries'], dict): - entries = downloads_obj['entries'] - else: - # Treat other keys except known settings as entries - for k, v in downloads_obj.items(): - if k in ('request_timeout_seconds', 'chunk_size_bytes'): - continue - entries[k] = v - - if args.download not in entries: - raise ValueError(f'Unknown download key for indicator {indicator}: {args.download}') - - selected = entries[args.download] - # If the entry is state-scoped, either expand templates for a single - # validated state or keep the templates when --state all is requested. - is_state_scoped = bool(selected.get('scope') == 'state' or 'relative_path_template' in selected or 'source_url_template' in selected) - if is_state_scoped: - rel_tmpl = selected.get('relative_path_template') - src_tmpl = selected.get('source_url_template') - if not rel_tmpl or not src_tmpl: - raise ValueError('State-scoped download entry must include "relative_path_template" and "source_url_template"') - if state_all_requested: - # keep templates un-expanded; main() will iterate over all states - raw_download_relative_path = rel_tmpl - source_url = src_tmpl - state_all_mode = True - else: - # require a validated single postal/fips for expansion - if not validated_postal or not validated_fips: - raise ValueError('This download key requires a valid --state/-s two-letter postal code to expand state-scoped templates') - try: - raw_download_relative_path = rel_tmpl.format(fips=validated_fips, postal=validated_postal) - source_url = src_tmpl.format(fips=validated_fips, postal=validated_postal) - except Exception as exc: - raise ValueError(f'Failed to expand state templates for download entry: {exc}') - else: - # Expect single-scope entry form - if 'relative_path' not in selected or 'source_url' not in selected: - raise ValueError('Download entry missing required "relative_path" or "source_url"') - - raw_download_relative_path = selected['relative_path'] - source_url = selected['source_url'] - - # Override timeout/chunk when supplied at the downloads level - if isinstance(downloads_obj.get('request_timeout_seconds'), int): - request_timeout_seconds = downloads_obj['request_timeout_seconds'] - if isinstance(downloads_obj.get('chunk_size_bytes'), int): - chunk_size_bytes = downloads_obj['chunk_size_bytes'] - - # If --download not provided, attempt to provide a clearer error message - if raw_download_relative_path is None or source_url is None: - # If an indicator JSON exists, inspect its downloads entries to give a helpful error. - if indicator_raw and isinstance(indicator_raw, dict) and isinstance(indicator_raw.get('downloads'), dict): - downloads_obj = indicator_raw['downloads'] - entries = {} - if 'entries' in downloads_obj and isinstance(downloads_obj['entries'], dict): - entries = downloads_obj['entries'] - else: - for k, v in downloads_obj.items(): - if k in ('request_timeout_seconds', 'chunk_size_bytes'): - continue - entries[k] = v - - if entries: - # If any entry is state-scoped, suggest using --download and --state - state_scoped = [k for k, v in entries.items() if (isinstance(v, dict) and (v.get('scope') == 'state' or 'relative_path_template' in v or 'source_url_template' in v))] - if state_scoped: - raise ValueError(f'Indicator {indicator!r} requires a download key and state for state-scoped entries. Try: --download {state_scoped[0]} --state ') - # Otherwise suggest available download keys - available = ', '.join(sorted(entries.keys())) - raise ValueError(f'Indicator {indicator!r} requires a --download key. Available keys: {available}') - - raise ValueError('Loaded indicator config does not expose expected download metadata for this slice') - - # The runner now lives in scripts/shared; resolve indicator folders - # relative to the repository `scripts/` directory (one level up). - SCRIPTS_ROOT = SCRIPTS_DIR.parent - if indicator != 'shared': - local_root = resolve_local_fn(SCRIPTS_ROOT / indicator) + state_all_mode = False + + if indicator == 'shared': + if not args.download: + raise ValueError('Shared fetch runs require --download to select a shared fetch output key') + + shared_config = _load_json_object(SCRIPTS_DIR / 'shared_config.json', 'shared') + request_timeout_seconds, chunk_size_bytes = _get_download_settings(shared_config, 'shared') + asset_name, version = _resolve_shared_fetch_target(shared_config, args.download, args.version) + # Build manifest for the requested environment; resolver accessors + # provide canonical roots so we don't need local/remote manifests here. + manifest = build_manifest.get_stage_manifest( + target_type='shared', + name=asset_name, + stage='fetch', + version=version, + environment=args.storage_mode, + ) + + output_key = _resolve_fetch_output_key(manifest['outputs'], args.download, f'shared asset {asset_name}') + manifest_output = manifest['outputs'][output_key] + raw_download_relative_path, source_url, state_all_mode = _resolve_fetch_source_and_relative( + manifest_output, + manifest_output['relative'], + validated_postal, + validated_fips, + state_all_requested, + ) + + # Obtain canonical roots from the resolver accessors instead of + # reading them from the JSON config directly. + local_root = resolve_path.get_shared_root(asset_name, version, 'local') + remote_root = resolve_path.get_shared_root(asset_name, version, 'remote') else: - local_root = resolve_local_fn(SCRIPTS_ROOT) + indicator_config = _load_json_object(scripts_root / indicator / f'{indicator}_config.json', f'indicator {indicator}') + versions = _require_mapping(indicator_config.get('versions'), f'{indicator}.versions') + version = _resolve_version(versions, f'indicator {indicator}', args.version) + request_timeout_seconds, chunk_size_bytes = _get_download_settings(indicator_config, indicator) + version_block = _require_mapping(versions.get(version), f'{indicator}.versions.{version}') + stages = _require_mapping(version_block.get('stages'), f'{indicator}.versions.{version}.stages') + fetch_stage = _require_mapping(stages.get('fetch'), f'{indicator}.versions.{version}.stages.fetch') + # fetch_outputs no longer needed here; manifest provides source metadata + # Obtain manifests for each environment so we can use the manifest + # roots instead of reading roots from the indicator config. + manifest = build_manifest.get_stage_manifest( + target_type='indicator', + name=indicator, + stage='fetch', + version=version, + environment=args.storage_mode, + ) + + output_key = _resolve_fetch_output_key(manifest['outputs'], args.download, f'indicator {indicator}') + manifest_output = manifest['outputs'][output_key] + raw_download_relative_path, source_url, state_all_mode = _resolve_fetch_source_and_relative( + manifest_output, + manifest_output['relative'], + validated_postal, + validated_fips, + state_all_requested, + ) - remote_root = getattr(raw_config, 'remote_root_path') + local_root = resolve_path.get_indicator_root(indicator, version, 'local') + remote_root = resolve_path.get_indicator_root(indicator, version, 'remote') return Config( indicator=indicator, + version=version, storage_mode=args.storage_mode, local_root_path=local_root, remote_root_path=remote_root, @@ -580,6 +639,7 @@ def main(argv=None) -> int: try: temp_cfg = Config( indicator=cfg.indicator, + version=cfg.version, storage_mode=cfg.storage_mode, local_root_path=cfg.local_root_path, remote_root_path=cfg.remote_root_path, diff --git a/scripts/shared/resolve_path.py b/scripts/shared/resolve_path.py new file mode 100644 index 0000000..3314220 --- /dev/null +++ b/scripts/shared/resolve_path.py @@ -0,0 +1,300 @@ +"""Centralized configuration-backed path resolver for indicators and shared assets. + +Public (flat) module-level functions: +- get_download_path(indicator: str, version: str, asset_key: str, environment: str = "local") -> dict +- get_dependency_version(indicator: str, version: str, dependency: str) -> str +- get_shared_asset_path(asset: str, version: str, category: str, asset_key: str | None = None, environment: str = "local") -> dict +- get_indicator_root(indicator: str, version: str, environment: str = "local") -> str +- get_shared_root(asset: str, version: str, environment: str = "local") -> str +- get_shared_version_block(config: dict, asset_name: str, version: str) -> dict + +Return contract: +- Functions that resolve locations return a dictionary containing at least the keys + "root" and "relative". +- When `environment == 'local'`, `root` is returned as an absolute path resolved + against the project root. When `environment == 'remote'`, `root` is the raw remote + root string (for example, an S3 URI). +- `environment` accepts only the values 'local' or 'remote' and will raise on + invalid input. + +Notes: +- The module exposes a flat functional interface suitable for use from R via + `reticulate` and from other callers. Internally a singleton `_PathResolver` + caches parsed JSON configs to minimize I/O. +""" + +from __future__ import annotations + +import json +import logging +from pathlib import Path + + +LOGGER = logging.getLogger(__name__) +LOGGER.addHandler(logging.NullHandler()) + +SCRIPTS_ROOT = Path(__file__).resolve().parent.parent +SHARED_CONFIG_PATH = Path(__file__).with_name("shared_config.json") +VALID_ENVIRONMENTS = {"local", "remote"} +VALID_SHARED_CATEGORIES = {"downloads", "preprocessed_input"} + + +class _PathResolver: + def __init__(self) -> None: + self._indicator_config_cache: dict[str, dict[str, object]] = {} + self._shared_config_cache: dict[str, object] | None = None + + def get_download_path( + self, + indicator: str, + version: str, + asset_key: str, + environment: str = "local", + ) -> dict[str, str]: + self._validate_environment(environment) + config = self._load_indicator_config(indicator) + version_block = self._get_indicator_version_block(config, indicator, version) + fetch_outputs = self._get_stage_outputs( + version_block, + f"{indicator}.versions.{version}", + "fetch", + ) + asset = self._require_mapping( + fetch_outputs.get(asset_key), + f"{indicator}.versions.{version}.stages.fetch.outputs.{asset_key}", + ) + relative = self._require_non_empty_string( + asset.get("relative_path"), + f"{indicator}.versions.{version}.stages.fetch.outputs.{asset_key}.relative_path", + ) + root = self._get_root(config, environment, f"indicator {indicator}") + return {"root": root, "relative": relative} + + def get_dependency_version(self, indicator: str, version: str, dependency: str) -> str: + config = self._load_indicator_config(indicator) + version_block = self._get_indicator_version_block(config, indicator, version) + required_shared_assets = self._require_mapping( + version_block.get("required_shared_assets"), + f"{indicator}.versions.{version}.required_shared_assets", + ) + return self._require_non_empty_string( + required_shared_assets.get(dependency), + f"{indicator}.versions.{version}.required_shared_assets.{dependency}", + ) + + def get_shared_asset_path( + self, + asset: str, + version: str, + category: str, + asset_key: str | None = None, + environment: str = "local", + ) -> dict[str, str]: + self._validate_environment(environment) + if category not in VALID_SHARED_CATEGORIES: + self._fail( + f"Invalid shared asset category {category!r}; expected one of {sorted(VALID_SHARED_CATEGORIES)}" + ) + + config = self._load_shared_config() + assets = self._require_mapping(config.get("assets"), "shared.assets") + asset_versions = self._require_mapping(assets.get(asset), f"shared.assets.{asset}") + version_block = self._require_mapping(asset_versions.get(version), f"shared.assets.{asset}.{version}") + + if category == "preprocessed_input": + preprocess_outputs = self._get_stage_outputs( + version_block, + f"shared.assets.{asset}.{version}", + "preprocess", + ) + preprocess_asset = self._require_mapping( + preprocess_outputs.get(asset), + f"shared.assets.{asset}.{version}.stages.preprocess.outputs.{asset}", + ) + relative = self._require_non_empty_string( + preprocess_asset.get("relative_path_template"), + f"shared.assets.{asset}.{version}.stages.preprocess.outputs.{asset}.relative_path_template", + ) + else: + if not asset_key: + self._fail( + "asset_key is required when resolving shared downloads" + ) + fetch_outputs = self._get_stage_outputs( + version_block, + f"shared.assets.{asset}.{version}", + "fetch", + ) + download_asset = self._require_mapping( + fetch_outputs.get(asset_key), + f"shared.assets.{asset}.{version}.stages.fetch.outputs.{asset_key}", + ) + relative = self._require_non_empty_string( + download_asset.get("relative_path_template"), + f"shared.assets.{asset}.{version}.stages.fetch.outputs.{asset_key}.relative_path_template", + ) + + root = self._get_root(config, environment, "shared") + return {"root": root, "relative": relative} + + def _load_indicator_config(self, indicator: str) -> dict[str, object]: + if indicator in self._indicator_config_cache: + return self._indicator_config_cache[indicator] + + config_path = SCRIPTS_ROOT / indicator / f"{indicator}_config.json" + config = self._load_json_object(config_path, f"indicator {indicator}") + self._indicator_config_cache[indicator] = config + return config + + def _load_shared_config(self) -> dict[str, object]: + if self._shared_config_cache is not None: + return self._shared_config_cache + + self._shared_config_cache = self._load_json_object(SHARED_CONFIG_PATH, "shared") + return self._shared_config_cache + + def _load_json_object(self, config_path: Path, label: str) -> dict[str, object]: + if not config_path.exists(): + self._fail(f"Missing {label} config file: {config_path}") + + LOGGER.info("Loading %s config from %s", label, config_path) + with config_path.open("r", encoding="utf-8") as handle: + payload = json.load(handle) + + if not isinstance(payload, dict): + self._fail(f"Expected {label} config to be a JSON object: {config_path}") + return payload + + def _get_indicator_version_block( + self, + config: dict[str, object], + indicator: str, + version: str, + ) -> dict[str, object]: + versions = self._require_mapping(config.get("versions"), f"{indicator}.versions") + return self._require_mapping(versions.get(version), f"{indicator}.versions.{version}") + + def _get_shared_version_block( + self, + config: dict[str, object], + asset_name: str, + version: str, + ) -> dict[str, object]: + assets = self._require_mapping(config.get("assets"), "shared.assets") + asset_versions = self._require_mapping(assets.get(asset_name), f"shared.assets.{asset_name}") + version_block = asset_versions.get(version) + if version_block is None: + self._fail(f"Version {version!r} not found for shared asset {asset_name}") + return self._require_mapping(version_block, f"shared.assets.{asset_name}.{version}") + + def _get_stage_outputs( + self, + version_block: dict[str, object], + field_prefix: str, + stage_name: str, + ) -> dict[str, object]: + stages = self._require_mapping(version_block.get("stages"), f"{field_prefix}.stages") + stage_block = self._require_mapping(stages.get(stage_name), f"{field_prefix}.stages.{stage_name}") + return self._require_mapping(stage_block.get("outputs"), f"{field_prefix}.stages.{stage_name}.outputs") + + def _get_root(self, config: dict[str, object], environment: str, label: str) -> str: + field_name = "local_root_path" if environment == "local" else "remote_root_path" + return self._require_non_empty_string(config.get(field_name), f"{label}.{field_name}") + + def _validate_environment(self, environment: str) -> None: + if environment not in VALID_ENVIRONMENTS: + self._fail(f"Invalid environment {environment!r}; expected one of {sorted(VALID_ENVIRONMENTS)}") + + def _require_mapping(self, value: object, field_name: str) -> dict[str, object]: + if not isinstance(value, dict): + self._fail(f"Expected {field_name} to be a JSON object") + return value + + def _require_non_empty_string(self, value: object, field_name: str) -> str: + if not isinstance(value, str) or not value: + self._fail(f"Expected {field_name} to be a non-empty string") + return value + + def _fail(self, message: str) -> None: + LOGGER.error(message) + raise ValueError(message) + + +_RESOLVER: _PathResolver | None = None + + +def _get_resolver() -> _PathResolver: + global _RESOLVER + if _RESOLVER is None: + _RESOLVER = _PathResolver() + return _RESOLVER + + +def get_download_path( + indicator: str, + version: str, + asset_key: str, + environment: str = "local", +) -> dict[str, str]: + return _get_resolver().get_download_path(indicator, version, asset_key, environment) + + +def get_dependency_version(indicator: str, version: str, dependency: str) -> str: + return _get_resolver().get_dependency_version(indicator, version, dependency) + + +def get_shared_asset_path( + asset: str, + version: str, + category: str, + asset_key: str | None = None, + environment: str = "local", +) -> dict[str, str]: + return _get_resolver().get_shared_asset_path(asset, version, category, asset_key, environment) + + +def get_indicator_root(indicator: str, version: str, environment: str = "local") -> str: + """Return the configured root path for an indicator. + + For `environment=='local'` this returns an absolute path resolved + against the project root. For `environment=='remote'` it returns + the raw remote root string from the config (e.g. an S3 URI). + """ + resolver = _get_resolver() + config = resolver._load_indicator_config(indicator) + # validate version exists + _ = resolver._get_indicator_version_block(config, indicator, version) + root = resolver._get_root(config, environment, f"indicator {indicator}") + if environment == "local": + # Resolve relative local roots against the project root (scripts parent) + project_root = SCRIPTS_ROOT.parent + return str((project_root / root).resolve()) + return root + + +def get_shared_root(asset: str, version: str, environment: str = "local") -> str: + """Return the configured root path for a shared asset. + + For `environment=='local'` this returns an absolute path resolved + against the project root. For `environment=='remote'` it returns + the raw remote root string from the shared config. + """ + resolver = _get_resolver() + config = resolver._load_shared_config() + # validate asset/version exist + _ = resolver._get_shared_version_block(config, asset, version) + root = resolver._get_root(config, environment, "shared") + if environment == "local": + project_root = SCRIPTS_ROOT.parent + return str((project_root / root).resolve()) + return root + + +def get_shared_version_block(config: dict[str, object], asset_name: str, version: str) -> dict[str, object]: + """Public helper to validate and return a shared asset's version block. + + This delegates to the internal resolver implementation so callers can + rely on a single authoritative implementation for shared-version + validation and error messages. + """ + return _get_resolver()._get_shared_version_block(config, asset_name, version) \ No newline at end of file diff --git a/scripts/shared/shared_config.json b/scripts/shared/shared_config.json index a6374f8..b8c2369 100644 --- a/scripts/shared/shared_config.json +++ b/scripts/shared/shared_config.json @@ -1,25 +1,86 @@ { - "local_root_path": "shared/pipeline/", - "remote_root_path": "s3://pedp-data-preserved/ejscreen-data-processing/shared/pipeline/", - "downloads_subtree_name": "downloads", - "preprocessed_input_subtree_name": "preprocessed_input", - "target_year": 2020, - "tiger_bg_relative_path_template": "downloads/tiger_lines/2020/bg/tl_2020_{fips}_bg.zip", - "census_block_weights_relative_path_template": "preprocessed_input/census_block_weights_2020/census_block_weights_2020_{postal}.csv", - "downloads": { + "local_root_path": "./pipeline/shared/", + "remote_root_path": "s3://pedp-data-preserved/ejscreen-data-processing/pipeline/shared/", + "download_settings": { "request_timeout_seconds": 120, - "chunk_size_bytes": 1048576, - "entries": { - "tiger_bg_2020": { - "scope": "state", - "source_url_template": "https://www2.census.gov/geo/tiger/TIGER2020/BG/tl_2020_{fips}_bg.zip", - "relative_path_template": "downloads/tiger_lines/2020/bg/tl_2020_{fips}_bg.zip" + "chunk_size_bytes": 1048576 + }, + "assets": { + "census_block_weights": { + "1.0": { + "stages": { + "preprocess": { + "outputs": { + "census_block_weights": { + "type": "shared_asset", + "scope": "single", + "relative_path_template": "census_block_weights/1.0/preprocessed_input/census_block_weights_2020_w2022pops.csv" + } + } + } }, - "tiger_block_2020": { - "scope": "state", - "source_url_template": "https://www2.census.gov/geo/tiger/TIGER2022/TABBLOCK20/tl_2022_{fips}_tabblock20.zip", - "relative_path_template": "downloads/tiger_lines/2022/tabblock20/tl_2022_{fips}_tabblock20.zip" + "notes": "These are EmmaLi's new-improved weights that include the acs_2022_bg_pop column to use for identifying zero-population block groups." + }, + "0.6": { + "stages": { + "preprocess": { + "outputs": { + "census_block_weights": { + "type": "shared_asset", + "scope": "single", + "relative_path_template": "census_block_weights/0.6/preprocessed_input/census_block_weights_2020.csv" + } + } + } + }, + "notes": "These are still EmmaLi's original weights, but we're going to read the whole-country file, not one postal code at a time." + }, + "0.5": { + "stages": { + "preprocess": { + "outputs": { + "census_block_weights": { + "type": "shared_asset", + "scope": "postal", + "relative_path_template": "census_block_weights/0.5/preprocessed_input/census_block_weights_2020_{postal}.csv" + } + } + } + }, + "notes": "These were EmmaLi's original weights, used up through her fix for issues #15 and #16." + } + }, + "tiger_bg": { + "2020": { + "stages": { + "fetch": { + "outputs": { + "tiger_bg_2020": { + "type": "download", + "scope": "state", + "source_url_template": "https://www2.census.gov/geo/tiger/TIGER2020/BG/tl_2020_{fips}_bg.zip", + "relative_path_template": "downloads/tiger_lines/2020/bg/tl_2020_{fips}_bg.zip" + } + } + } + } + } + }, + "tiger_block": { + "2022": { + "stages": { + "fetch": { + "outputs": { + "tiger_block_2020": { + "type": "download", + "scope": "state", + "source_url_template": "https://www2.census.gov/geo/tiger/TIGER2022/TABBLOCK20/tl_2022_{fips}_tabblock20.zip", + "relative_path_template": "downloads/tiger_lines/2022/tabblock20/tl_2022_{fips}_tabblock20.zip" + } + } + } + } } } } -} +} \ No newline at end of file diff --git a/scripts/test_harness/test_build_manifest.R b/scripts/test_harness/test_build_manifest.R new file mode 100644 index 0000000..6949e7a --- /dev/null +++ b/scripts/test_harness/test_build_manifest.R @@ -0,0 +1,78 @@ +#!/usr/bin/env Rscript +# test_build_manifest.R +# +# Dry-run harness for R users to exercise the Python `build_manifest` +# via `reticulate` and inspect the returned named-list +# +# Usage examples: (run from the scripts/ folder): +# Rscript test_harness/test_build_manifest.R --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0 +# Rscript test_harness/test_build_manifest.R --storage-mode remote --target-type indicator --name o3 --stage score --version 1.0 +# Rscript test_harness/test_build_manifest.R --storage-mode local --target-type shared --name tiger_bg --stage fetch --version 2020 +# Rscript test_harness/test_build_manifest.R --storage-mode local --target-type shared --name census_block_weights --stage preprocess --version 1.0 + +# checking package requirements +if (!requireNamespace("reticulate", quietly = TRUE)) { + stop("Please install the 'reticulate' R package to run this harness.") +} +if (!requireNamespace("jsonlite", quietly = TRUE)) { + stop("Please install the 'jsonlite' R package to format output (install.packages('jsonlite')).") +} +if (!requireNamespace("optparse", quietly = TRUE)) { + stop("Please install the 'optparse' R package to format output (install.packages('optparse')).") +} + +# load in packages for test harness: +library(reticulate) +library(jsonlite) +library(optparse) + +# pulling in args from the command line: +args <- commandArgs(trailingOnly = TRUE) + +option_list <- list( + make_option(c('--storage-mode'), type='character', + default='local', required = T, dest = "storage_mode"), + make_option(c('--target-type'), required = T, + dest = "target_type"), + make_option(c('--name'), required = T), + make_option(c('--stage'), required = T), + make_option(c('--version'), required = T)) + +parser <- OptionParser(usage='%prog [options]', option_list=option_list) +args_all <- parse_args(parser, positional_arguments = FALSE) +opts <- args_all + +# read in build manifest +shared_path <- "./shared" +py_build_manifest <- NULL +try({ + py_build_manifest <- import_from_path("build_manifest", path = shared_path) +}, silent = TRUE) + +# if we can pull in resolve path, run checks: +if (is.null(py_build_manifest)) { + cat("ERROR: failed to import 'build_manifest' from", shared_path, "\n") + quit(status = 1) +} + +# try it out! +build_manifrst_from_args <- function(opts) { + parts <- "NULL - did not run" + parts <- tryCatch({ + py_build_manifest$get_stage_manifest( + target_type = opts$target_type, + name = opts$name, + stage = opts$stage, + version = opts$version, + environment = opts$storage_mode + ) + }, error = function(e) { + cat("get_download_path failed:", e$message, "\n") + }) + cat("\n--- Resolver checks ---\n") + # return! + return(parts) +} + +result <- build_manifrst_from_args(opts) +cat(toJSON(result, pretty = TRUE, auto_unbox = TRUE), "\n") diff --git a/scripts/test_harness/test_build_manifest.py b/scripts/test_harness/test_build_manifest.py new file mode 100644 index 0000000..ffb9f81 --- /dev/null +++ b/scripts/test_harness/test_build_manifest.py @@ -0,0 +1,62 @@ +"""Dry-run harness for the shared build_manifest module. + +Examples (run from the `scripts/` folder): + + python3 test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0 + python3 test_harness/test_build_manifest.py --storage-mode remote --target-type indicator --name o3 --stage score --version 1.0 + python3 test_harness/test_build_manifest.py --storage-mode local --target-type shared --name tiger_bg --stage fetch --version 2020 + python3 test_harness/test_build_manifest.py --storage-mode local --target-type shared --name census_block_weights --stage preprocess --version 1.0 +""" + +from __future__ import annotations + +import argparse +import json +import logging +import sys +from pathlib import Path + + +LOGGER = logging.getLogger(__name__) + +SHARED_SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "shared" +if str(SHARED_SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SHARED_SCRIPTS_DIR)) + +import build_manifest # type: ignore[import-not-found] + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Dry-run stage manifest build for an indicator or shared asset.") + parser.add_argument("--storage-mode", dest="storage_mode", required=True, choices=("local","remote")) + parser.add_argument("--target-type", required=True) + parser.add_argument("--name", required=True) + parser.add_argument("--stage", required=True) + parser.add_argument("--version", required=True) + return parser + + +def main() -> int: + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") + parser = _build_parser() + args = parser.parse_args() + + try: + manifest = build_manifest.get_stage_manifest( + target_type=args.target_type, + name=args.name, + stage=args.stage, + version=args.version, + environment=args.storage_mode, + ) + except Exception as exc: + LOGGER.error("Manifest dry-run failed: %s", exc) + return 1 + + payload = {"status": "DRY_RUN_MANIFEST_BUILT", **manifest} + print(json.dumps(payload, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) \ No newline at end of file diff --git a/scripts/test_harness/test_resolve_path.R b/scripts/test_harness/test_resolve_path.R new file mode 100644 index 0000000..25fb12c --- /dev/null +++ b/scripts/test_harness/test_resolve_path.R @@ -0,0 +1,111 @@ +#!/usr/bin/env Rscript +# test_resolve_path.R +# +# Dry-run harness for R users to exercise the Python `resolve_path` +# via `reticulate` and inspect the returned named-list +# +# Usage examples: (run from the scripts/ folder): +# Rscript test_harness/test_resolve_path.R --storage-mode local --type indicator --name o3 --version 1.0 --key raw_o3 +# Rscript test_harness/test_resolve_path.R --storage-mode remote --type indicator --name o3 --version 1.0 --key raw_o3 +# Rscript test_harness/test_resolve_path.R --storage-mode local --type shared --name tiger_bg --version 2020 --category downloads --key tiger_bg_2020 +# Rscript test_harness/test_resolve_path.R --storage-mode remote --type shared --name census_block_weights --version 1.0 --category preprocessed_input + +# checking package requirements +if (!requireNamespace("reticulate", quietly = TRUE)) { + stop("Please install the 'reticulate' R package to run this harness.") +} +if (!requireNamespace("jsonlite", quietly = TRUE)) { + stop("Please install the 'jsonlite' R package to format output (install.packages('jsonlite')).") +} +if (!requireNamespace("optparse", quietly = TRUE)) { + stop("Please install the 'optparse' R package to format output (install.packages('optparse')).") +} + +# load in packages for test harness: +library(reticulate) +library(jsonlite) +library(optparse) + +# pulling in args from the command line: +args <- commandArgs(trailingOnly = TRUE) + +option_list <- list( + make_option(c('--storage-mode'), type='character', + default='local', required = T, dest = "storage_mode"), + make_option(c('--type'), type='character', default=NULL, required = T), + make_option(c('--name'), type='character', default=NULL, required = T), + make_option(c('--version'), type='character', default=NULL, required = T), + make_option(c('--key'), type='character', default=NULL), + make_option(c('--category'), type='character', default=NULL) +) + +parser <- OptionParser(usage='%prog [options]', option_list=option_list) +args_all <- parse_args(parser, positional_arguments = FALSE) +opts <- args_all + +# checks for args: +if (opts$type == "indicator"){ + if (is.null(opts$key)) { + message("--key is required when --type is indicator") + quit(status = 1) + } + if (!is.null(opts$category)){ + message("--category is not valid when --type is indicator") + quit(status = 1) + } +} else { + if(is.null(opts$category)){ + message("--category is required when --type shared") + quit(status = 1) + } + if(opts$category == "downloads" && is.null(opts$key)){ + message("--key is required when resolving shared downloads") + quit(status = 1) + } + if(opts$category == "preprocessed_input" & !is.null(opts$key)){ + message("--key is not valid for shared preprocessed_input resolution") + quit(status = 1) + } +} + +# read in resolve path: +shared_path <- "./shared" +py_resolve = NULL +try({ + py_resolve <- import_from_path("resolve_path", path = shared_path) +}, silent = TRUE) + +# if we can pull in resolve path, run checks: +if (is.null(py_resolve)) { + cat("ERROR: failed to import 'resolve_path' from", shared_path, "\n") + quit(status = 1) +} + +resolve_from_args <- function(opts) { + parts <- "NULL - did not run" + # if we're working with an indicator, get the download path + if (opts$type == "indicator") { + parts <- tryCatch({ + py_resolve$get_download_path(opts$name, + opts$version, + opts$key, + opts$storage_mode) + }, error = function(e) { + cat("get_download_path failed:", e$message, "\n") + }) + cat("\n--- Resolver checks ---\n") + # if we're working with a shared asset, get the shared asset path: + } else { + parts <- py_resolve$get_shared_asset_path(asset = opts$name, + version = opts$version, + category = opts$category, + asset_key = opts$key, + environment = opts$storage_mode) + } + + # return! + return(parts) +} + +result <- resolve_from_args(opts) +cat(toJSON(result, pretty = TRUE, auto_unbox = TRUE), "\n") diff --git a/scripts/test_harness/test_resolve_path.py b/scripts/test_harness/test_resolve_path.py new file mode 100644 index 0000000..a6395ef --- /dev/null +++ b/scripts/test_harness/test_resolve_path.py @@ -0,0 +1,81 @@ +"""Dry-run harness for the shared resolve_path module. + +Examples (run from the `scripts/` folder): + + python3 test_harness/test_resolve_path.py --storage-mode local --type indicator --name o3 --version 1.0 --key raw_o3 + python3 test_harness/test_resolve_path.py --storage-mode remote --type indicator --name o3 --version 1.0 --key raw_o3 + python3 test_harness/test_resolve_path.py --storage-mode local --type shared --name tiger_bg --version 2020 --category downloads --key tiger_bg_2020 + python3 test_harness/test_resolve_path.py --storage-mode remote --type shared --name census_block_weights --version 1.0 --category preprocessed_input +""" + +from __future__ import annotations + +import argparse +import json +import logging +import sys +from pathlib import Path + + +LOGGER = logging.getLogger(__name__) + +SHARED_SCRIPTS_DIR = Path(__file__).resolve().parents[1] / "shared" +if str(SHARED_SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SHARED_SCRIPTS_DIR)) + +import resolve_path # type: ignore[import-not-found] + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Dry-run path resolution for indicator and shared assets.") + parser.add_argument("--storage-mode", dest="storage_mode", required=True, choices=("local","remote")) + parser.add_argument("--type", required=True, choices=["indicator", "shared"]) + parser.add_argument("--name", required=True) + parser.add_argument("--version", required=True) + parser.add_argument("--key") + parser.add_argument("--category") + return parser + + +def _resolve_from_args(args: argparse.Namespace) -> dict[str, str]: + if args.type == "indicator": + if not args.key: + raise ValueError("--key is required when --type indicator") + if args.category is not None: + raise ValueError("--category is not valid when --type indicator") + return resolve_path.get_download_path(args.name, args.version, args.key, args.storage_mode) + + if args.category is None: + raise ValueError("--category is required when --type shared") + if args.category == "downloads" and not args.key: + raise ValueError("--key is required when resolving shared downloads") + if args.category == "preprocessed_input" and args.key is not None: + raise ValueError("--key is not valid for shared preprocessed_input resolution") + + return resolve_path.get_shared_asset_path( + asset=args.name, + version=args.version, + category=args.category, + asset_key=args.key, + environment=args.storage_mode, + ) + + +def main() -> int: + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s") + parser = _build_parser() + args = parser.parse_args() + + try: + resolved = _resolve_from_args(args) + except Exception as exc: + LOGGER.error("Resolver dry-run failed: %s", exc) + return 1 + + payload = {"status": "DRY_RUN_RESOLVED", **resolved} + print(json.dumps(payload, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) \ No newline at end of file diff --git a/scripts/test_harness/test_resolve_path_reticulate.R b/scripts/test_harness/test_resolve_path_reticulate.R new file mode 100644 index 0000000..4512abe --- /dev/null +++ b/scripts/test_harness/test_resolve_path_reticulate.R @@ -0,0 +1,87 @@ +#!/usr/bin/env Rscript +# test_resolve_path_reticulate.R +# +# Dry-run harness for R users to exercise the Python `resolve_path` and +# `build_manifest` modules via `reticulate` and inspect the returned named-list +# shapes. This is intended as a lightweight check you can run locally before +# wiring calls into production R scripts. +# +# Usage examples: +# Rscript scripts/test_harness/test_resolve_path_reticulate.R # runs both checks +# Rscript scripts/test_harness/test_resolve_path_reticulate.R --resolver # resolver checks only +# Rscript scripts/test_harness/test_resolve_path_reticulate.R --manifest # manifest check only +# +args <- commandArgs(trailingOnly = TRUE) +run_resolver <- TRUE +run_manifest <- TRUE +if (length(args) > 0) { + run_resolver <- any(grepl("--resolver", args)) || (!any(grepl("--resolver|--manifest", args)) ) + run_manifest <- any(grepl("--manifest", args)) || (!any(grepl("--resolver|--manifest", args)) ) +} + +if (!requireNamespace("reticulate", quietly = TRUE)) { + stop("Please install the 'reticulate' R package to run this harness.") +} +if (!requireNamespace("jsonlite", quietly = TRUE)) { + stop("Please install the 'jsonlite' R package to format output (install.packages('jsonlite')).") +} + +library(reticulate) +library(jsonlite) + +cat("Starting R reticulate resolver/manifest harness\n") + +shared_path <- "./scripts/shared" +py_resolve <- NULL +py_build_manifest <- NULL +try({ + py_resolve <- import_from_path("resolve_path", path = shared_path) +}, silent = TRUE) +try({ + py_build_manifest <- import_from_path("build_manifest", path = shared_path) +}, silent = TRUE) + +if (is.null(py_resolve)) { + cat("ERROR: failed to import 'resolve_path' from", shared_path, "\n") +} else if (run_resolver) { + cat("\n--- Resolver checks ---\n") + tryCatch({ + parts <- py_resolve$get_download_path("o3", "1.0", "raw_o3", "local") + cat("get_download_path('o3', '1.0', 'raw_o3', 'local') returned:\n") + cat(toJSON(parts, pretty = TRUE, auto_unbox = TRUE), "\n") + }, error = function(e) { + cat("get_download_path failed:", e$message, "\n") + }) + + tryCatch({ + dep_ver <- py_resolve$get_dependency_version("o3", "1.0", "census_block_weights") + cat("get_dependency_version('o3','1.0','census_block_weights') -> ", dep_ver, "\n") + shared_parts <- py_resolve$get_shared_asset_path("census_block_weights", dep_ver, "preprocessed_input", environment = "local") + cat("get_shared_asset_path('census_block_weights', , 'preprocessed_input') returned:\n") + cat(toJSON(shared_parts, pretty = TRUE, auto_unbox = TRUE), "\n") + }, error = function(e) { + cat("shared asset resolution failed:", e$message, "\n") + }) +} + +if (is.null(py_build_manifest)) { + cat("\nNOTE: failed to import 'build_manifest' from", shared_path, "\n") +} else if (run_manifest) { + cat("\n--- Manifest check ---\n") + tryCatch({ + manifest <- py_build_manifest$get_stage_manifest( + target_type = "indicator", + name = "o3", + stage = "preprocess", + version = "1.0", + environment = "local" + ) + cat("get_stage_manifest('indicator','o3','preprocess','1.0','local') returned keys:\n") + cat(paste(names(manifest), collapse = ", "), "\n") + cat(toJSON(manifest, pretty = TRUE, auto_unbox = TRUE), "\n") + }, error = function(e) { + cat("get_stage_manifest failed:", e$message, "\n") + }) +} + +cat("\nHarness complete.\n") diff --git a/scripts/utilities/validation/compareScores.py b/scripts/utilities/validation/compareScores.py new file mode 100644 index 0000000..13c41fe --- /dev/null +++ b/scripts/utilities/validation/compareScores.py @@ -0,0 +1,434 @@ +#!/usr/bin/env python3 +"""compareScores.py + +Slice 1 implementation: config + CLI merge, validation, --dry-run, and logging. + +This script intentionally implements only the configuration handling and validation +for the first slice. The actual comparison and plotting are implemented in later +slices. +""" +from __future__ import annotations + +import argparse +import json +import logging +import sys +from pathlib import Path +from typing import Any, Dict, Optional + +import math + +import numpy as np +import pandas as pd +import matplotlib.pyplot as plt +import textwrap +import geopandas as gpd +from matplotlib.cm import ScalarMappable +from matplotlib.colors import Normalize, TwoSlopeNorm + + +REQUIRED_FIELDS = ( + 'indicator', + 'state', + 'file_a', + 'id_a', + 'score_a', + 'version_a', + 'file_b', + 'id_b', + 'score_b', + 'version_b', +) + + +def load_config(path: Path) -> Dict[str, Any]: + try: + with path.open('r', encoding='utf-8') as fh: + return json.load(fh) + except Exception as exc: + raise RuntimeError(f'Failed to load config {path}: {exc}') from exc + + +def merge_config(file_config: Optional[Dict[str, Any]], cli_args: argparse.Namespace) -> Dict[str, Any]: + cfg: Dict[str, Any] = {} + if file_config: + cfg.update(file_config) + + # Map CLI names to config keys + cli_to_key = { + 'indicator': 'indicator', + 'state': 'state', + 'file_a': 'file_a', + 'id_a': 'id_a', + 'score_a': 'score_a', + 'version_a': 'version_a', + 'file_b': 'file_b', + 'id_b': 'id_b', + 'score_b': 'score_b', + 'version_b': 'version_b', + 'out_dir': 'out_dir', + } + + for arg_name, key in cli_to_key.items(): + val = getattr(cli_args, arg_name, None) + if val is not None: + cfg[key] = val + + # Normalize common path strings to plain strings + for pkey in ('file_a', 'file_b', 'out_dir'): + if pkey in cfg and isinstance(cfg[pkey], str): + cfg[pkey] = cfg[pkey].strip() + + return cfg + + +def validate_config(cfg: Dict[str, Any]) -> tuple[bool, list[str]]: + missing = [f for f in REQUIRED_FIELDS if not cfg.get(f)] + return (len(missing) == 0, missing) + + +def init_logging(script_dir: Path) -> None: + log_path = script_dir / 'compareScores.log' + handler = logging.FileHandler(log_path, mode='a', encoding='utf-8') + logging.basicConfig( + level=logging.INFO, + format='%(asctime)s %(levelname)s: %(message)s', + datefmt='%Y-%m-%d %H:%M:%S', + handlers=[handler], + ) + logging.info('=== compareScores log started ===') + + +def pretty_print_config(cfg: Dict[str, Any]) -> None: + print(json.dumps(cfg, indent=2, sort_keys=True)) + + +def pretty_print_cli_style_config(cfg: Dict[str, Any]) -> None: + """Print a user-facing config with underscore keys converted to hyphen keys. + + This is intended for dry-run output so the displayed option names match the + CLI hyphenated form (e.g. --version-a) while internal logging continues to + use underscore keys. + """ + cli_cfg = { (k.replace('_', '-')): v for k, v in cfg.items() } + print(json.dumps(cli_cfg, indent=2, sort_keys=True)) + + +def default_out_dir(state: str, indicator: str, version_a: str, version_b: str) -> str: + return str(Path('output') / state / 'compare' / f'{indicator}_{version_a}_vs_{version_b}') + + +def parse_args(argv: Optional[list[str]] = None) -> argparse.Namespace: + parser = argparse.ArgumentParser(description='Compare two indicator score CSVs (slice 1: config/dry-run)') + parser.add_argument('--config', type=str, default=None, help='Path to JSON config file (optional)') + + parser.add_argument('--indicator', type=str, help='Indicator slug (used for labels and filenames)') + parser.add_argument('--state', type=str, help='Two-letter state postal code (required)') + + parser.add_argument('--file-a', type=str, help='Path to CSV file A') + parser.add_argument('--id-a', type=str, help='ID column name in file A') + parser.add_argument('--score-a', type=str, help='Score column name in file A') + parser.add_argument('--version-a', type=str, help='Version label for file A') + + parser.add_argument('--file-b', type=str, help='Path to CSV file B') + parser.add_argument('--id-b', type=str, help='ID column name in file B') + parser.add_argument('--score-b', type=str, help='Score column name in file B') + parser.add_argument('--version-b', type=str, help='Version label for file B') + + parser.add_argument('--out-dir', type=str, default=None, help='Optional output directory') + parser.add_argument('--dry-run', action='store_true', help='Validate and print merged config; do not run') + + return parser.parse_args(argv) + + +def read_csv_coerce(path: str, id_col: str, score_col: str) -> pd.DataFrame: + df = pd.read_csv(path, dtype=str) + if id_col not in df.columns: + raise KeyError(f"ID column '{id_col}' not found in {path}") + if score_col not in df.columns: + raise KeyError(f"Score column '{score_col}' not found in {path}") + df[id_col] = df[id_col].astype(str).str.strip() + df[score_col] = pd.to_numeric(df[score_col], errors='coerce') + return df[[id_col, score_col]].copy() + + +def compute_stats(df: pd.DataFrame, score_a: str, score_b: str) -> Dict[str, float]: + valid = df[[score_a, score_b]].dropna() + diffs = (valid[score_a] - valid[score_b]).astype(float) + abs_diffs = diffs.abs() + mean_abs = float(abs_diffs.mean()) if len(abs_diffs) > 0 else float('nan') + median_abs = float(abs_diffs.median()) if len(abs_diffs) > 0 else float('nan') + rmse = float(np.sqrt(np.nanmean((valid[score_a] - valid[score_b]) ** 2))) if len(valid) > 0 else float('nan') + try: + pearson = float(valid[score_a].corr(valid[score_b])) if len(valid) > 1 else float('nan') + except Exception: + pearson = float('nan') + return { + 'matched_rows': int(len(df)), + 'rows_used': int(len(valid)), + 'dropped': int(len(df) - len(valid)), + 'mean_abs_diff': mean_abs, + 'median_abs_diff': median_abs, + 'rmse': rmse, + 'pearson': pearson, + } + + +def plot_scatter(df: pd.DataFrame, score_a: str, score_b: str, out_path: Path, stats: Dict[str, float], title: str) -> None: + fig, ax = plt.subplots(figsize=(7, 7)) + ax.scatter(df[score_a], df[score_b], alpha=0.6, s=20, edgecolors='none') + if not df.empty: + mins = np.nanmin([df[score_a].min(), df[score_b].min()]) + maxs = np.nanmax([df[score_a].max(), df[score_b].max()]) + else: + mins, maxs = 0.0, 1.0 + pad = (maxs - mins) * 0.02 if maxs != mins else 0.5 + ax.plot([mins - pad, maxs + pad], [mins - pad, maxs + pad], color='red', linestyle='--', linewidth=1) + ax.set_xlabel(score_a) + ax.set_ylabel(score_b) + ax.set_title(title) + ax.set_aspect('equal', adjustable='box') + + stats_txt = textwrap.dedent(f""" + Matched rows: {stats['matched_rows']} + Rows used: {stats['rows_used']} + Dropped: {stats['dropped']} + Mean abs diff: {stats['mean_abs_diff']:.6g} + Median abs diff: {stats['median_abs_diff']:.6g} + RMSE: {stats['rmse']:.6g} + Pearson r: {stats['pearson']:.6g} + """) + props = dict(boxstyle='round', facecolor='white', alpha=0.8) + ax.text(0.02, 0.98, stats_txt, transform=ax.transAxes, fontsize=9, va='top', ha='left', bbox=props) + + out_path.parent.mkdir(parents=True, exist_ok=True) + fig.tight_layout() + fig.savefig(out_path, dpi=150) + plt.close(fig) + + +def _resolve_scripts_dir() -> Path: + current_path = Path(__file__).resolve() + for parent in current_path.parents: + if parent.name == 'scripts': + return parent + raise RuntimeError(f'Unable to locate scripts directory from {current_path}') + + +def _read_block_groups_geodataframe(tiger_zip_path: Path) -> gpd.GeoDataFrame: + candidates = [str(tiger_zip_path)] + if tiger_zip_path.suffix.lower() == '.zip': + candidates.append(f'zip://{tiger_zip_path.as_posix()}') + + last_err = None + for candidate in dict.fromkeys(candidates): + try: + gdf = gpd.read_file(candidate) + return gdf + except Exception as exc: + last_err = exc + raise RuntimeError(f'Failed to read block-group data from {tiger_zip_path}: {last_err}') + + +def prepare_map_and_plot(merged_df: pd.DataFrame, state: str, out_dir: Path, indicator: str, version_a: str, version_b: str) -> None: + # Derive FIPS from first matched geoid (first 2 digits) + if merged_df.empty: + raise RuntimeError('No matched rows to map') + sample_id = merged_df['id'].astype(str).iloc[0] + fips = sample_id[:2] + + # Resolve TIGER path relative to scripts/shared/pipeline/downloads/... + scripts_dir = _resolve_scripts_dir() + tiger_rel = Path('downloads') / 'tiger_lines' / '2020' / 'bg' / f'tl_2020_{fips}_bg.zip' + tiger_path = scripts_dir / 'shared' / 'pipeline' / tiger_rel + if not tiger_path.exists(): + raise FileNotFoundError(f'TIGER block-group ZIP not found: {tiger_path}') + + bg_gdf = _read_block_groups_geodataframe(tiger_path) + if 'GEOID' not in bg_gdf.columns: + raise RuntimeError(f"Expected 'GEOID' column in TIGER block-group data: {tiger_path}") + if 'geometry' not in bg_gdf.columns: + raise RuntimeError(f"Expected 'geometry' column in TIGER block-group data: {tiger_path}") + + map_df = merged_df.copy() + map_df['id'] = map_df['id'].astype(str).str.strip() + + bg_gdf['GEOID'] = bg_gdf['GEOID'].astype(str).str.strip() + bg_plot = bg_gdf.merge(map_df[['id', 'score_a', 'score_b', 'score_diff']], left_on='GEOID', right_on='id', how='left') + state_outline = bg_gdf.dissolve() + + # Compute scales + score_values = pd.concat([map_df['score_a'], map_df['score_b']], ignore_index=True).dropna() + diff_values = map_df['score_diff'].dropna() + score_scale_max = float(score_values.max()) if not score_values.empty else 0.0 + diff_max_abs = float(diff_values.abs().max()) if not diff_values.empty else 0.0 + + score_scale_bound = score_scale_max if score_scale_max > 0 else 1.0 + diff_scale_bound = diff_max_abs if diff_max_abs > 1.0 else 1.0 + score_norm = Normalize(vmin=0.0, vmax=score_scale_bound) + diff_norm = TwoSlopeNorm(vmin=-diff_scale_bound, vcenter=0.0, vmax=diff_scale_bound) + + fig = plt.figure(figsize=(15, 16), constrained_layout=True) + outer_grid = fig.add_gridspec(2, 1, height_ratios=[1, 1.12]) + top_grid = outer_grid[0].subgridspec(1, 2, wspace=0.04) + bottom_grid = outer_grid[1].subgridspec(1, 3, width_ratios=[1, 1, 0.9], wspace=0.18) + + ax_ejam = fig.add_subplot(top_grid[0, 0]) + ax_new = fig.add_subplot(top_grid[0, 1]) + ax_diff = fig.add_subplot(bottom_grid[0, :2]) + ax_scatter = fig.add_subplot(bottom_grid[0, 2]) + + def _plot_map_panel(state_outline, bg_plot, column_name, cmap, norm, ax, title): + bg_plot.plot( + column=column_name, + cmap=cmap, + norm=norm, + linewidth=0.05, + edgecolor='#b8b8b8', + legend=False, + missing_kwds={'color': '#b3b3b3'}, + ax=ax, + ) + state_outline.boundary.plot(ax=ax, color='#4a4a4a', linewidth=0.5) + ax.set_title(title) + ax.set_axis_off() + + _plot_map_panel(state_outline, bg_plot, 'score_a', 'Reds', score_norm, ax_ejam, f'{indicator} {version_a}') + _plot_map_panel(state_outline, bg_plot, 'score_b', 'Reds', score_norm, ax_new, f'{indicator} {version_b}') + _plot_map_panel(state_outline, bg_plot, 'score_diff', 'RdBu', diff_norm, ax_diff, 'Difference (A - B)') + + # Scatter panel + plot_scatter(map_df.rename(columns={'score_a': 'score_a', 'score_b': 'score_b'}), 'score_a', 'score_b', out_dir / f'map_scatter_{indicator}_{version_a}_vs_{version_b}.png', compute_stats(map_df, 'score_a', 'score_b'), f'{indicator} scatter') + + score_colorbar = fig.colorbar(ScalarMappable(norm=score_norm, cmap='Reds'), ax=[ax_ejam, ax_new], fraction=0.03, pad=0.02) + score_colorbar.set_label('Score value') + diff_colorbar = fig.colorbar(ScalarMappable(norm=diff_norm, cmap='RdBu'), ax=ax_diff, fraction=0.03, pad=0.02) + diff_colorbar.set_label('Score difference (A - B)') + + fig.suptitle(f'{state}: Block-group score comparison', fontsize=15, y=0.98) + map_out = out_dir / f'{state}_map_{indicator}_{version_a}_vs_{version_b}.png' + fig.savefig(map_out, dpi=150, bbox_inches='tight') + plt.close(fig) + logging.info('Map written to %s', map_out) + + +def main(argv: Optional[list[str]] = None) -> int: + args = parse_args(argv) + script_dir = Path(__file__).resolve().parent + init_logging(script_dir) + + file_cfg = None + if args.config: + cfg_path = Path(args.config) + if not cfg_path.exists(): + print(f'ERROR: config file not found: {cfg_path}', file=sys.stderr) + logging.error('Config file not found: %s', cfg_path) + return 2 + try: + file_cfg = load_config(cfg_path) + except Exception as exc: + print(f'ERROR: {exc}', file=sys.stderr) + logging.exception('Failed to load config') + return 3 + + merged = merge_config(file_cfg, args) + + # If out_dir unspecified, propose a default (do not create yet) + if 'out_dir' not in merged or not merged.get('out_dir'): + if merged.get('state') and merged.get('indicator') and merged.get('version_a') and merged.get('version_b'): + merged['out_dir'] = default_out_dir(merged['state'], merged['indicator'], merged['version_a'], merged['version_b']) + + valid, missing = validate_config(merged) + if not valid: + print('ERROR: missing required configuration values:', file=sys.stderr) + for m in missing: + print(f' - {m}', file=sys.stderr) + logging.error('Validation failed; missing: %s', missing) + return 4 + + logging.info('Merged run configuration: %s', json.dumps(merged)) + + if args.dry_run: + print('Dry-run: merged configuration (no files will be read or written)') + pretty_print_cli_style_config(merged) + logging.info('Dry-run completed') + return 0 + + # For slice 1 we stop after validation and reporting; later slices will perform reading and comparison. + print('Configuration validated. Running comparator (slice 2)') + + # Prepare output paths + out_dir = Path(merged['out_dir']) + out_dir.mkdir(parents=True, exist_ok=True) + + matched_csv = out_dir / f"matched_rows_{merged['indicator']}_{merged['version_a']}_vs_{merged['version_b']}.csv" + scatter_png = out_dir / f"scatter_{merged['indicator']}_{merged['version_a']}_vs_{merged['version_b']}.png" + compare_log = out_dir / 'compare.log' + + try: + # Read inputs + if not Path(merged['file_a']).exists(): + raise FileNotFoundError(f"File A not found: {merged['file_a']}") + if not Path(merged['file_b']).exists(): + raise FileNotFoundError(f"File B not found: {merged['file_b']}") + + df_a = read_csv_coerce(merged['file_a'], merged['id_a'], merged['score_a']) + df_b = read_csv_coerce(merged['file_b'], merged['id_b'], merged['score_b']) + + # Normalize join column name + df_a = df_a.rename(columns={merged['id_a']: 'id', merged['score_a']: 'score_a'}) + df_b = df_b.rename(columns={merged['id_b']: 'id', merged['score_b']: 'score_b'}) + + # Inner join + merged_df = df_a.merge(df_b, on='id', how='inner') + merged_df['score_diff'] = merged_df['score_a'] - merged_df['score_b'] + + # Write matched CSV + merged_df.to_csv(matched_csv, index=False) + + # Compute stats and plot scatter + stats = compute_stats(merged_df, 'score_a', 'score_b') + axis_label_a = f"{merged['score_a']} ({merged['version_a']})" + axis_label_b = f"{merged['score_b']} ({merged['version_b']})" + plot_title = f"{merged['indicator']}: {merged['version_a']} vs {merged['version_b']}" + # For plotting, use columns renamed to score_a/score_b but label axes with original names + plot_df = merged_df.rename(columns={'score_a': axis_label_a, 'score_b': axis_label_b}) + plot_scatter(plot_df, axis_label_a, axis_label_b, scatter_png, stats, plot_title) + + # Write compare summary log + with compare_log.open('w', encoding='utf-8') as fh: + fh.write('Comparison summary\n') + fh.write(f"matched_rows={stats['matched_rows']}\n") + fh.write(f"rows_used={stats['rows_used']}\n") + fh.write(f"dropped={stats['dropped']}\n") + fh.write(f"mean_abs_diff={stats['mean_abs_diff']:.6g}\n") + fh.write(f"median_abs_diff={stats['median_abs_diff']:.6g}\n") + fh.write(f"rmse={stats['rmse']:.6g}\n") + fh.write(f"pearson={stats['pearson']:.6g}\n") + + logging.info('Comparison completed: matched=%d rows_used=%d', stats['matched_rows'], stats['rows_used']) + print(f'Wrote matched rows CSV: {matched_csv}') + print(f'Wrote scatter plot PNG: {scatter_png}') + print(f'Wrote compare summary: {compare_log}') + + # Attempt to produce the four-panel map (slice 3). If TIGER data is missing + # or plotting fails, log the error but do not fail the whole run. + try: + prepare_map_and_plot(merged_df, merged['state'], out_dir, merged['indicator'], merged['version_a'], merged['version_b']) + except FileNotFoundError as fnf: + print(f'Map not produced: {fnf}') + logging.warning('Map not produced: %s', fnf) + except Exception as exc_map: + print(f'Warning: map generation failed: {exc_map}') + logging.exception('Map generation failed') + + except Exception as exc: + print(f'ERROR: {exc}', file=sys.stderr) + logging.exception('Comparison failed') + return 5 + + return 0 + + +if __name__ == '__main__': + raise SystemExit(main()) diff --git a/scripts/utilities/validation/compare_o3_RI_v0.6_ejam.json b/scripts/utilities/validation/compare_o3_RI_v0.6_ejam.json new file mode 100644 index 0000000..a0441d7 --- /dev/null +++ b/scripts/utilities/validation/compare_o3_RI_v0.6_ejam.json @@ -0,0 +1,13 @@ +{ + "indicator": "o3", + "state": "RI", + "file_a": "../../../pipeline/o3/v0.6/output/indicators/RI/final_bg_scores.csv", + "id_a": "block_group_geoid", + "score_a": "o3_score", + "version_a": "v0.6", + "file_b": "../../../scripts/utilities/validation/output/RI/ejam_o3_subset.csv", + "id_b": "ejam_uniq_id", + "score_b": "o3", + "version_b": "ejam", + "out_dir": "./output/RI/compare/o3_v0.6_vs_ejam/" +} diff --git a/scripts/utilities/validation/compare_o3_WY_v0.6_ejam.json b/scripts/utilities/validation/compare_o3_WY_v0.6_ejam.json new file mode 100644 index 0000000..3957dcd --- /dev/null +++ b/scripts/utilities/validation/compare_o3_WY_v0.6_ejam.json @@ -0,0 +1,13 @@ +{ + "indicator": "o3", + "state": "WY", + "file_a": "../../../pipeline/o3/v0.6/output/indicators/WY/final_bg_scores.csv", + "id_a": "block_group_geoid", + "score_a": "o3_score", + "version_a": "v0.6", + "file_b": "../../../scripts/utilities/validation/output/WY/ejam_o3_subset.csv", + "id_b": "ejam_uniq_id", + "score_b": "o3", + "version_b": "ejam", + "out_dir": "./output/WY/compare/o3_v0.6_vs_ejam/" +} diff --git a/scripts/utilities/validation/compare_o3_v0.6_v0.5.json b/scripts/utilities/validation/compare_o3_v0.6_v0.5.json new file mode 100644 index 0000000..de35718 --- /dev/null +++ b/scripts/utilities/validation/compare_o3_v0.6_v0.5.json @@ -0,0 +1,13 @@ +{ + "indicator": "o3", + "state": "MT", + "file_a": "../../../pipeline/o3/v0.6/output/indicators/MT/final_bg_scores.csv", + "id_a": "block_group_geoid", + "score_a": "o3_score", + "version_a": "v0.6", + "file_b": "../../../pipeline/o3/v0.5/output/indicators/MT/final_bg_scores.csv", + "id_b": "block_group_geoid", + "score_b": "o3_score", + "version_b": "v0.5", + "out_dir": "./output/MT/compare/o3_v0.6_vs_v0.5/" +} diff --git a/scripts/utilities/validation/examples/compare_config_example.json b/scripts/utilities/validation/examples/compare_config_example.json new file mode 100644 index 0000000..6dc7f62 --- /dev/null +++ b/scripts/utilities/validation/examples/compare_config_example.json @@ -0,0 +1,13 @@ +{ + "indicator": "o3", + "state": "MT", + "file_a": "./pipeline/o3/v0.6/output/indicators/MT/final_bg_scores.csv", + "id_a": "block_group_geoid", + "score_a": "o3_score", + "version_a": "v0.6", + "file_b": "./pipeline/o3/v0.5/output/indicators/MT/final_bg_scores.csv", + "id_b": "block_group_geoid", + "score_b": "o3_score", + "version_b": "v0.5", + "out_dir": "./output/MT/compare/o3_v0.6_vs_v0.5/" +}