fix: apply user-defined aws_batch_tags regardless of BATCH_EMIT_TAGS#3210
fix: apply user-defined aws_batch_tags regardless of BATCH_EMIT_TAGS#3210odncode wants to merge 4 commits into
Conversation
User-defined tags (aws_batch_tags and METAFLOW_BATCH_DEFAULT_TAGS) were trapped inside the 'if BATCH_EMIT_TAGS:' guard, which controls Metaflow's internal observability tags. This caused user tags to be silently dropped when BATCH_EMIT_TAGS was not set. In enterprise deployments with mandatory AWS tagging policies (SCPs), this results in Batch job submission failures with no indication that the root cause is a missing Metaflow config flag. Fix: move the aws_batch_tags block outside the BATCH_EMIT_TAGS guard. Internal tags remain gated; user-defined tags are always applied. Includes an AST-based regression test that verifies aws_batch_tags is not referenced inside any BATCH_EMIT_TAGS conditional block. Closes Netflix#3209
Greptile SummaryThis PR decouples user-defined
Confidence Score: 5/5Safe to merge — the change is a minimal, targeted indentation shift with a clear mechanical justification and a new regression test. The production change is a single block moved out of a conditional; the logic for building and iterating over No files require special attention. Important Files Changed
Reviews (4): Last reviewed commit: "style: add missing EOF newline (pre-comm..." | Re-trigger Greptile |
Addresses review feedback: previous test only checked the negative (tags not inside guard) without verifying they exist at all. New test ensures aws_batch_tags is referenced in create_job, guarding against silent removal of the feature.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3210 +/- ##
=========================================
Coverage ? 28.85%
=========================================
Files ? 381
Lines ? 52467
Branches ? 9260
=========================================
Hits ? 15138
Misses ? 36315
Partials ? 1014 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Signed-off-by: odncode <nnajiodera2@gmail.com>
Summary
Fixes #3209
User-defined
aws_batch_tagsandMETAFLOW_BATCH_DEFAULT_TAGSare currently dropped silently whenBATCH_EMIT_TAGSis not set. This PR ensures that user-defined tags are always respected and applied to AWS Batch jobs, regardless of internal telemetry settings.Problem
The
aws_batch_tagsblock inbatch.py(L401-403) sits inside theif BATCH_EMIT_TAGS:guard (L380).While
BATCH_EMIT_TAGSis intended to toggle Metaflow's internal observability tags (such asapp=metaflowormetaflow.flow_name), it inadvertently gates user-defined tags meant for cost attribution or compliance.In enterprise environments with mandatory AWS Service Control Policies (SCPs) or AWS Config rules requiring specific tags, this causes Batch job submissions to fail with access-denied errors unless
BATCH_EMIT_TAGSis enabled.Fix
aws_batch_tagsprocessing block outside of theif BATCH_EMIT_TAGS:guard.Testing
test/unit/test_batch_user_tags.py— an AST-based structural test that parsescreate_joband verifiesaws_batch_tagsis not referenced inside anyif BATCH_EMIT_TAGS:block.PR Checklist
Fixes #3209)Closes #3209