Draft PR of Data Versioning code -- partial implementation only so far

scripts/R_programmer_guide.md

@@ -0,0 +1,135 @@
+# R Programmer Guide: Using the Python Resolver and Manifest Builder
+
+This guide shows how R users can call the shared Python utilities (`resolve_path` and
+`build_manifest`) from R via the `reticulate` package. It includes examples, common
+patterns, and links to the bundled test harnesses you can run to verify return shapes
+before wiring them into your pipelines.
+
+Location of test harnesses
+- `scripts/test_harness/test_resolve_path.py` — dry-run resolver outputs (indicator/shared)
+- `scripts/test_harness/test_build_manifest.py` — dry-run manifest builder outputs
+
+Note: Python dictionaries returned by these modules map to named lists in R when
+imported via `reticulate`. Access elements with the `$` operator.
+
+---
+
+## 1. Using `resolve_path` from R
+
+Purpose: obtain `root` and `relative` path components for indicator downloads or shared
+assets so your R ingestion code can be environment-agnostic.
+
+Example (reticulate):
+
+```R
+library(reticulate)
+resolve_path <- import_from_path("resolve_path", path = "./scripts/shared")
+
+# Simple download lookup (indicator)
+parts <- resolve_path$get_download_path("o3", "1.0", "raw_o3", "local")
+print(parts$root)      # e.g. "./pipeline/o3/"
+print(parts$relative)  # e.g. "v1.0/downloads/..."
+
+# Shared asset lookup (preprocessed template). No interpolation is performed by resolver.
+shared_ver <- resolve_path$get_dependency_version("o3", "1.0", "census_block_weights")
+shared_parts <- resolve_path$get_shared_asset_path("census_block_weights", shared_ver, "preprocessed_input", environment = "local")
+print(shared_parts$relative) # contains token like "...{postal}..."
+
+```
+
+Token replacement and building a usable path:
+
+```R
+state_postal <- "WY"
+relative_filled <- gsub("\\{postal\\}", state_postal, shared_parts$relative)
+full_path <- paste0(shared_parts$root, relative_filled)
+
+# Read locally
+df <- readr::read_csv(full_path)
+
+# Or, for remote S3 URIs, stream using an S3-aware reader (do not use `file.path`)
+```
+
+Quick CLI checks (shell) to confirm what `reticulate` will receive:
+
+```bash
+python scripts/test_harness/test_resolve_path.py --storage-mode local --type indicator --name o3 --version 1.0 --key raw_o3
+python scripts/test_harness/test_resolve_path.py --storage-mode local --type shared --name census_block_weights --version 1.0 --category preprocessed_input
+```
+
+These produce a small JSON object with `root` and `relative` fields; the harness adds a
+`status` field for clarity.
+
+---
+
+## 2. Using `build_manifest` from R
+
+Purpose: retrieve a compiled stage manifest (inputs/outputs) that already includes
+`root` and `relative` entries for each declared input and output. This is useful when
+you want a single manifest-driven object to drive multiple parts of a workflow.
+
+Example (reticulate):
+
+```R
+library(reticulate)
+build_manifest <- import_from_path("build_manifest", path = "./scripts/shared")
+
+manifest <- build_manifest$get_stage_manifest(
+  target_type = "indicator",
+  name = "o3",
+  stage = "preprocess",
+  version = "1.0",
+  environment = "local"
+)
+
+# Inspect the returned named list
+print(names(manifest))          # typically: "inputs" "outputs"
+print(manifest$inputs$primary_o3$root)
+print(manifest$outputs$main_tract_averages$relative)
+
+```
+
+Quick CLI check (shell) to confirm manifest shape and contents:
+
+```bash
+python scripts/test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0
+```
+
+The harness prints a JSON object prefixed with `status` that mirrors the manifest
+structure you will get via `reticulate`.
+
+Notes and recommendations
+- The manifest entries for inputs/outputs already include `root` + `relative` so you
+  can construct `full_path <- paste0(entry$root, entry$relative)` in R.
+- If an entry's `relative` contains tokens (e.g. `{postal}`), perform R-side
+  substitution with `gsub()` before concatenation.
+- Avoid using `file.path()` for S3 URIs — prefer string concatenation to preserve
+  forward slashes.
+
+---
+
+## 3. Where to run tests and what to expect
+
+- Resolver test harness: `scripts/test_harness/test_resolve_path.py`
+  - Use to validate `get_download_path` and `get_shared_asset_path` return shapes
+  - Example: `python scripts/test_harness/test_resolve_path.py --storage-mode local --type indicator --name o3 --version 1.0 --key raw_o3`
+
+- Manifest test harness: `scripts/test_harness/test_build_manifest.py`
+  - Use to validate `get_stage_manifest` returns `inputs`/`outputs` with compiled entries
+  - Example: `python scripts/test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0`
+
+I added a short R script in `scripts/test_harness/` that calls these functions via
+`reticulate` and prints the R named-list shapes. Run it with `Rscript` to see what
+your `reticulate` calls should return:
+
+```bash
+Rscript scripts/test_harness/test_resolve_path_reticulate.R
+```
+
+The script runs a small set of resolver and manifest checks and prints JSON-formatted
+outputs to stdout. It is intended as a low-risk local verification step before you
+embed `reticulate` calls in production R code.
+
+---
+
+Last updated: 2026-06-13

scripts/agToDo.md

@@ -0,0 +1,36 @@
+# Checklist: Configuration & Path Resolution Refactor
+
+## Phase 1: Environment & Schema Setup
+- [x] Create a new Git branch for the refactor: `dataVersioning`
+- [x] Update `scripts/o3/o3_config.json` to match the new schema structure (including `local_root_path`, `remote_root_path`, and stage definitions for inputs/outputs)
+- [x] Update `scripts/shared/shared_config.json` to match the new global asset schema
+
+## Phase 2: Core Path Resolution (The Machinist)
+- [x] Prompt AI to generate `scripts/shared/resolve_path.py` based on Section 1 specifications
+- [x] Prompt AI to generate the CLI test harness `test_resolve_path.py` based on Section 2 specifications
+- [x] Test `resolve_path.py` using the harness against `o3_config.json`:
+    - [x] Verify local resolution returns accurate `{"root": ..., "relative": ...}` dictionary
+    - [x] Verify remote resolution handles S3 paths correctly without Windows backslash errors
+- [x] Test `resolve_path.py` using the harness against `shared_config.json` (e.g., checking dependency version resolution logic)
+
+## Phase 3: Workflow Mapping (The Architect)
+- [x] Prompt AI to generate `scripts/shared/build_manifest.py` based on the manifest specification
+- [x] Prompt AI to generate the CLI test harness `test_manifest.py` 
+- [x] Test `build_manifest.py` using the harness for the `o3` `preprocess` stage:
+    - [x] Verify that it auto-resolves global shared dependencies (e.g., pulling correct version of `census_block_weights`)
+    - [x] Verify that auxiliary output folders are cleanly mapped in the final dictionary
+
+## Phase 4: Tracer Bullet Integration (O3 Testing)
+- [x] Refactor the `fetch_raw` script to consume paths via `resolve_path.py`
+- [x] Refactor `scripts/o3/o3_preprocess.py` (Python) to pull its execution map via `build_manifest.py`
+- [x] Refactor `scripts/o3/o3_score.py` (Python or R) to pull its execution map via `build_manifest.py`
+    - [x] Verify full run locally
+    - [x] Verify full run remotely (dry-run or execution on S3 via `fsspec`/R cloud tools)
+
+## Phase 5: Prepare for and merge with `indicators` branch
+- [ ] Migrate the remaining indicator configuration files (`pm25_config.json`, etc.) to the new schema
+- [ ] Run the `test_manifest.py` harness against all newly migrated indicators to catch syntax/schema errors early
+- [ ] Update execution scripts for all remaining indicators to consume the new `build_manifest` workflow
+- [ ] change `xx_indictator.py` name to `xx_score.py` as each indicator's code is upgraded to data versioning
+- [ ] Run final end-to-end testing across the entire pipeline universe
+- [ ] Merge branch into `indicators`

scripts/build_manifest_specification.md

@@ -0,0 +1,185 @@
+# Module Specification: build_manifest.py
+**Purpose:** Provide a high-level orchestration planning tool shared across Python and R runtimes via reticulate. It reads either an indicator's or a shared asset's stage-specific configuration block, coordinates with `resolve_path.py`, and returns a single unified "Run Manifest" dictionary containing every file path required for that execution phase.
+
+---
+
+## 1. System Requirements & Context
+- **Language Compatibility:** Pure Python 3.x using only standard libraries (`json`, `os`, `pathlib`). No third-party dependencies.
+- **Interoperability:** Must expose a flat functional interface at the module root level to support smooth R `reticulate` execution ergonomics. Under the hood, it must import and utilize `resolve_path.py` to obtain physical storage vectors.
+- **Directory Context:** Assumes execution occurs from the `project_root/` directory.
+- **Module Location:** The manifest module will live at `./scripts/shared/build_manifest.py`.
+
+---
+
+## 2. Core Functional Requirements (The "What")
+
+The module acts as a workflow compiler. Instead of operational scripts parsing raw JSON trees to discover their inputs, this module processes the configuration schema to produce an actionable bill of materials.
+
+### 1. Configuration Target Structure
+The module must support two target domains:
+
+- **Indicator target:** Read `./scripts/{name}/{name}_config.json`, navigate to `versions -> {version} -> stages -> {stage}`.
+- **Shared target:** Read `./scripts/shared/shared_config.json`, navigate to `assets -> {name} -> {version} -> stages -> {stage}`.
+
+In both cases, the module must parse the targeted stage block's explicit `inputs` and `outputs` mappings. A stage may omit `inputs`; in that case, the returned manifest must contain an empty `inputs` dictionary.
+
+### 2. Dependency Tracking Invariant
+When an entry under a stage's `inputs` section points to a global shared asset (e.g., `type: "shared_asset"`), `build_manifest.py` must automatically step back up the configuration tree, query `resolve_path.py` to identify the precise version string pinned under `required_shared_assets` for that indicator version, and then pass that information along to get the asset's shared paths. The calling processing script should never have to explicitly declare or look up dependency versions.
+
+Indicator-owned fetched assets are now modeled as outputs of the indicator's `fetch` stage. Shared downloaded assets are modeled as outputs of the shared asset's `fetch` stage. Shared preprocessed assets are modeled as outputs of the shared asset's `preprocess` stage.
+
+This dependency-version lookup rule applies to indicator targets. Shared targets are first-class manifest targets themselves and must be compilable directly from `shared_config.json` without being routed through an indicator config.
+
+### 3. Output Data Shape Contract
+The function must compile and return a **single Python Dictionary** containing exactly two top-level keys: `"inputs"` and `"outputs"`. 
+
+The value of every single nested asset identity key within this manifest must follow the resolver contract and include at least the keys shown below:
+```python
+{
+    "root": "string base path",
+    "relative": "string versioned relative path (or template)"
+}
+
+```
+
+---
+
+## 3. Required Module Interface (Exposed Function)
+
+### `get_stage_manifest(target_type: str, name: str, stage: str, version: str, environment: str = "local") -> dict`
+
+`target_type` must accept only `"indicator"` or `"shared"`, and invalid values must fail fast with a clear error.
+
+`environment` must accept only `"local"` or `"remote"`, and invalid values must fail fast with a clear error.
+
+#### Expected Behavior:
+
+1. **Load Configuration:**
+- If `target_type="indicator"`, read and parse `./scripts/{name}/{name}_config.json`.
+- If `target_type="shared"`, read and parse `./scripts/shared/shared_config.json`.
+2. **Isolate Stage Requirements:**
+- If `target_type="indicator"`, navigate to `["versions"][version]["stages"][stage]`.
+- If `target_type="shared"`, navigate to `["assets"][name][version]["stages"][stage]`.
+
+If the target, stage, or version does not exist, raise an explicit descriptive error.
+3. **Compile Inputs:** Loop through all items defined inside the stage's `inputs` block:
+- For indicator targets:
+    - If an input maps to an indicator fetch asset, request its coordinates from `resolve_path.get_download_path()`, which resolves through the indicator's `fetch` stage outputs.
+    - If an input maps to a global shared asset, query `resolve_path.get_dependency_version()` to find the required version, then pass that version to `resolve_path.get_shared_asset_path()`.
+    - If an input maps to an indicator-owned `file`, `directory`, or `template`, compile it directly against the indicator root path using its `relative_path`.
+- For shared targets:
+    - If the stage defines no `inputs`, return an empty `inputs` dictionary.
+    - If shared-stage inputs are later defined, they must be compiled strictly from their declared entry types. The implementation must not invent fallback rules for undeclared or transitional shared input shapes.
+
+
+4. **Compile Outputs:** Loop through all items defined inside the stage's `outputs` block. Match them against indicator paths or directory structures, compiling them into the identical dual-key dictionary output format.
+* For indicator targets, compile outputs against the indicator root path using `relative_path`.
+* For shared targets, compile outputs against the shared root path using the output entry's declared relative path field. Template tokens such as `{postal}` or `{fips}` must remain intact.
+5. **Consolidate and Return:** Return the unified manifest layout.
+
+---
+
+## 4. Manifest Dictionary Structural Contract (Target Output Shape)
+
+The AI code generator must guarantee that calling this module results in a dictionary nested exactly like the sample structure below:
+
+```python
+{
+    "inputs": {
+        "primary_asset_name": {
+            "root": "./pipeline/o3/",
+            "relative": "v1.0/downloads/2020/2020_ozone_daily_8hour_maximum.txt.gz"
+        },
+        "dependent_shared_asset_name": {
+            "root": "./pipeline/shared/",
+            "relative": "census_block_weights/1.0/preprocessed_input/census_block_weights_2020_{postal}.csv"
+        }
+    },
+    "outputs": {
+        "output_directory": {
+            "root": "./pipeline/o3/",
+            "relative": "v1.0/preprocessed_input/"
+        },
+        "main_tract_averages": {
+            "root": "./pipeline/o3/",
+            "relative": "v1.0/preprocessed_input/o3_tract_annual_average.csv"
+        }
+    }
+}
+
+```
+
+Shared-target example:
+
+```python
+{
+    "inputs": {},
+    "outputs": {
+        "tiger_bg_2020": {
+            "root": "./pipeline/shared/",
+            "relative": "downloads/tiger_lines/2020/bg/tl_2020_{fips}_bg.zip"
+        }
+    }
+}
+
+```
+
+---
+
+## 5. Implementation Verification / Test Cases
+
+1. **Local Preprocess Context Resolution:**
+`get_stage_manifest(target_type="indicator", name="o3", stage="preprocess", version="1.0", environment="local")`
+=> Must return an input dictionary mapping both internal indicator assets and external `census_block_weights` location dictionaries with local file-system roots.
+2. **Remote Score Context Resolution:**
+`get_stage_manifest(target_type="indicator", name="o3", stage="score", version="1.0", environment="remote")`
+=> Must return an input/output mapping dictionary prefixing all `"root"` parameters with the explicit S3 protocol block retrieved from the configuration file, keeping relative string structures preserved for pipeline orchestration.
+3. **Local Shared Fetch Context Resolution:**
+`get_stage_manifest(target_type="shared", name="tiger_bg", stage="fetch", version="2020", environment="local")`
+=> Must return an outputs dictionary rooted at the shared local path with the unexpanded state-scoped relative template preserved.
+4. **Local Shared Preprocess Context Resolution:**
+`get_stage_manifest(target_type="shared", name="census_block_weights", stage="preprocess", version="1.0", environment="local")`
+=> Must return the currently declared preprocess outputs for that shared asset with the shared local root and the relative template preserved.
+
+### Command-Line Dry-Run Validation Set
+
+
+Assuming the current working directory is `scripts/`, the following harness commands should validate the current implementation surface:
+
+```bash
+python3 test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage preprocess --version 1.0
+```
+
+```bash
+python3 test_harness/test_build_manifest.py --storage-mode remote --target-type indicator --name o3 --stage score --version 1.0
+```
+
+```bash
+python3 test_harness/test_build_manifest.py --storage-mode local --target-type indicator --name o3 --stage fetch --version 1.0
+```
+
+```bash
+python3 test_harness/test_build_manifest.py --storage-mode local --target-type shared --name tiger_bg --stage fetch --version 2020
+```
+
+```bash
+python3 test_harness/test_build_manifest.py --storage-mode local --target-type shared --name census_block_weights --stage preprocess --version 1.0
+```
+
+Negative fast-fail checks:
+
+```bash
+python3 test_harness/test_build_manifest.py --storage-mode staging --target-type indicator --name o3 --stage preprocess --version 1.0
+```
+
+```bash
+python3 test_harness/test_build_manifest.py --storage-mode local --target-type nonsense --name o3 --stage preprocess --version 1.0
+```
+
+## 6. Current Strictness Decision
+
+`build_manifest.py` should remain strict and should not implement fallback logic for incomplete or transitional config shapes.
+
+Shared assets are first-class manifest targets and must be supported directly by the manifest interface.
+
+`census_block_weights` is still an evolving shared-asset configuration surface. Until its final stage layout is settled, the manifest implementation should only honor the stages and fields explicitly present in config, without adding compatibility layers or silent fallbacks.