Skip to content

Commit b6a6580

Browse files
nooarcheclaude
andcommitted
Add EXQ-008/009 experiments + runner --loop mode for continuous operation
New experiment scripts (smoke-tested): - experiments/candidate_count_ablation.py (EXQ-008, MECH-063) Tests whether E3 trajectory selection over multiple candidates is load-bearing. Conditions: SINGLE(n=1), STANDARD(n=32), EXPANDED(n=128). - experiments/residue_weight_sensitivity.py (EXQ-009, MECH-062) Tests whether rho_residue routing weight is load-bearing. Conditions: RHO_ZERO(0.0), RHO_LOW(0.1), RHO_STANDARD(0.5), RHO_HIGH(2.0). Runner improvements: - experiment_runner.py: --loop / --loop-interval flags for continuous operation. After queue exhaustion, polls experiment_queue.json every N seconds (default 60) for newly-added items. Lets the runner stay alive between sessions. - Initial queue display now prunes already-completed items on startup. Queue updates (experiment_queue.json): - EXQ-008 (MECH-063, high priority) - EXQ-009 (MECH-062, high priority) - EXQ-010..013 (low priority, extended-seed re-runs of MECH-059/056/061/060 PASS results for confidence accumulation — keeps runner busy between new experiments) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 9991835 commit b6a6580

4 files changed

Lines changed: 925 additions & 39 deletions

File tree

experiment_queue.json

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,102 @@
107107
"estimated_runtime_min": 22,
108108
"status": "pending",
109109
"status_reason": "Script written and smoke-tested 2026-02-27. Addresses EXQ-004 FAIL root cause: atomic GridWorld has no multi-step action structure for completion gate to protect. MacroStepGrid with MACRO_LENGTH=4."
110+
},
111+
{
112+
"queue_id": "EXQ-008",
113+
"backlog_id": "EVB-0036",
114+
"claim_id": "MECH-063",
115+
"title": "E3 Candidate Count Ablation",
116+
"description": "Tests whether the number of E2 trajectory candidates available to E3 is load-bearing for harm avoidance. Three conditions: SINGLE (1 candidate, no real selection), STANDARD (32, default), EXPANDED (128, wider search). PASS: STANDARD < SINGLE (selection matters) AND EXPANDED <= STANDARD*1.10 (no collapse at high counts).",
117+
"script": "experiments/candidate_count_ablation.py",
118+
"args": ["--episodes", "200", "--seeds", "7", "42", "99"],
119+
"seeds": 3,
120+
"conditions": 3,
121+
"episodes_per_run": 200,
122+
"estimated_runtime_min": 22,
123+
"priority": "high",
124+
"status": "pending",
125+
"status_reason": "Script written 2026-02-28. Tests MECH-063 (trajectory_selection.candidate_count_load_bearing). Uses standard GridWorld with num_candidates override on generate_trajectories()."
126+
},
127+
{
128+
"queue_id": "EXQ-009",
129+
"backlog_id": "EVB-0032",
130+
"claim_id": "MECH-062",
131+
"title": "Residue Routing Weight Sensitivity",
132+
"description": "Tests whether the E3 residue routing weight (rho_residue) is load-bearing. Four conditions: RHO_ZERO (rho=0, agent is ethically amnesic), RHO_LOW (0.1), RHO_STANDARD (0.5, default), RHO_HIGH (2.0). PASS: RHO_STANDARD < RHO_ZERO (residue routing helps) AND RHO_HIGH <= RHO_ZERO*1.20 (high weight doesn't catastrophically destabilise).",
133+
"script": "experiments/residue_weight_sensitivity.py",
134+
"args": ["--episodes", "200", "--seeds", "7", "42", "99"],
135+
"seeds": 3,
136+
"conditions": 4,
137+
"episodes_per_run": 200,
138+
"estimated_runtime_min": 30,
139+
"priority": "high",
140+
"status": "pending",
141+
"status_reason": "Script written 2026-02-28. Tests MECH-062 (residue.routing_weight_load_bearing). Sets config.e3.rho_residue per condition. Key claim: residue field must have positive routing weight to influence trajectory selection."
142+
},
143+
{
144+
"queue_id": "EXQ-010",
145+
"backlog_id": "EVB-0037",
146+
"claim_id": "MECH-059",
147+
"title": "Control Plane Precision Separation (extended seeds)",
148+
"description": "Extended-seed re-run of EXQ-000 (MECH-059 PASS). Adds seeds 13, 77, 123, 256, 314, 999 for tighter confidence intervals. Same MERGED vs SEPARATED conditions. Contributes to P1 evidence accumulation for the precision channel separation claim.",
149+
"script": "experiments/control_plane_precision_separation.py",
150+
"args": ["--episodes", "200", "--seeds", "13", "77", "123", "256", "314", "999"],
151+
"seeds": 6,
152+
"conditions": 2,
153+
"episodes_per_run": 200,
154+
"estimated_runtime_min": 30,
155+
"priority": "low",
156+
"status": "pending",
157+
"status_reason": "Low-priority re-run for confidence accumulation. EXQ-000 PASS (seeds 7,42,99). Adds 6 more seeds."
158+
},
159+
{
160+
"queue_id": "EXQ-011",
161+
"backlog_id": "EVB-0039",
162+
"claim_id": "MECH-056",
163+
"title": "Residue Trajectory Placement (extended seeds)",
164+
"description": "Extended-seed re-run of EXQ-001 (MECH-056 PASS). Adds seeds 13, 77, 123, 256, 314, 999. TRAJECTORY-WIDE vs ENDPOINT-ONLY conditions. Builds replication depth for the residue placement claim.",
165+
"script": "experiments/residue_trajectory_placement.py",
166+
"args": ["--episodes", "200", "--seeds", "13", "77", "123", "256", "314", "999"],
167+
"seeds": 6,
168+
"conditions": 2,
169+
"episodes_per_run": 200,
170+
"estimated_runtime_min": 30,
171+
"priority": "low",
172+
"status": "pending",
173+
"status_reason": "Low-priority re-run for confidence accumulation. EXQ-001 PASS (seeds 7,42,99). Adds 6 more seeds."
174+
},
175+
{
176+
"queue_id": "EXQ-012",
177+
"backlog_id": "EVB-0041",
178+
"claim_id": "MECH-061",
179+
"title": "Commitment Boundary Token Reclassification (extended seeds)",
180+
"description": "Extended-seed re-run of EXQ-003 (MECH-061 PASS). Adds seeds 13, 77, 123, 256, 314, 999. PRE_COMMIT vs POST_COMMIT channel correlation conditions. Tightens confidence on commit boundary separation.",
181+
"script": "experiments/commitment_boundary_validation.py",
182+
"args": ["--episodes", "200", "--seeds", "13", "77", "123", "256", "314", "999"],
183+
"seeds": 6,
184+
"conditions": 2,
185+
"episodes_per_run": 200,
186+
"estimated_runtime_min": 30,
187+
"priority": "low",
188+
"status": "pending",
189+
"status_reason": "Low-priority re-run for confidence accumulation. EXQ-003 PASS (seeds 7,42,99). Adds 6 more seeds."
190+
},
191+
{
192+
"queue_id": "EXQ-013",
193+
"backlog_id": "EVB-0043",
194+
"claim_id": "MECH-060",
195+
"title": "Write-Locus Contamination (extended seeds)",
196+
"description": "Extended-seed re-run of EXQ-005 (MECH-060 PASS). Adds seeds 13, 77, 123, 256, 314, 999. FULL vs CONTAMINATED_DURABLE vs CONTAMINATED_RESIDUE. Builds replication depth on the write-locus separation result.",
197+
"script": "experiments/write_locus_contamination.py",
198+
"args": ["--episodes", "200", "--seeds", "13", "77", "123", "256", "314", "999"],
199+
"seeds": 6,
200+
"conditions": 3,
201+
"episodes_per_run": 200,
202+
"estimated_runtime_min": 45,
203+
"priority": "low",
204+
"status": "pending",
205+
"status_reason": "Low-priority re-run for confidence accumulation. EXQ-005 PASS (seeds 7,42,99). Adds 6 more seeds."
110206
}
111207
]
112208
}

experiment_runner.py

Lines changed: 106 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -296,6 +296,23 @@ def main():
296296
action="store_true",
297297
help="Show what would run without executing"
298298
)
299+
parser.add_argument(
300+
"--loop",
301+
action="store_true",
302+
help=(
303+
"After exhausting the queue, poll experiment_queue.json every "
304+
"--loop-interval seconds for new items instead of exiting. "
305+
"Lets the runner stay alive so clicking Start in the explorer "
306+
"keeps it running without a manual restart."
307+
),
308+
)
309+
parser.add_argument(
310+
"--loop-interval",
311+
type=int,
312+
default=60,
313+
metavar="SECONDS",
314+
help="Seconds between queue re-checks in --loop mode (default: 60)",
315+
)
299316
args = parser.parse_args()
300317

301318
status_path = args.status_file or find_default_status_path()
@@ -358,54 +375,104 @@ def handle_signal(sig, frame):
358375
return
359376

360377
print(f"[runner] PID {os.getpid()} — ready to run {len(items)} queued experiments", flush=True)
378+
if args.loop:
379+
print(f"[runner] Loop mode: will poll queue every {args.loop_interval}s after exhaustion", flush=True)
361380

362381
completed_ids = {c["queue_id"] for c in existing_completed}
363382

364-
for item in items:
365-
queue_id = item["queue_id"]
366-
367-
# Skip already completed
368-
if queue_id in completed_ids:
369-
print(f"[runner] Skipping {queue_id} ({item['claim_id']}) — already completed", flush=True)
370-
continue
371-
372-
# Skip items with no script
373-
script = REPO_ROOT / item["script"]
374-
if not script.exists():
375-
print(f"[runner] Skipping {queue_id} ({item['claim_id']}) — script not found: {item['script']}", flush=True)
376-
# Update queue item status in live status file
377-
for qi in status["queue"]:
378-
if qi["queue_id"] == queue_id:
379-
qi["status"] = "needs_script"
380-
write_status(status, status_path)
381-
continue
383+
# Prune any already-completed items from the initial queue display
384+
status["queue"] = [qi for qi in status["queue"] if qi["queue_id"] not in completed_ids]
385+
write_status(status, status_path)
382386

383-
# Run it
384-
result = run_experiment(item, status, status_path, calibration)
387+
while True:
388+
ran_any = False
385389

386-
# Move to completed
387-
completed_entry = {
388-
"queue_id": queue_id,
389-
"backlog_id": item.get("backlog_id", ""),
390-
"claim_id": item.get("claim_id", ""),
391-
"title": item.get("title", ""),
392-
"description": item.get("description", ""),
393-
"result": result["result"],
394-
"result_summary": result["result_summary"],
395-
"completed_at": result["completed_at"],
396-
"output_file": result.get("output_file", ""),
397-
}
398-
status["completed"].append(completed_entry)
399-
completed_ids.add(queue_id)
390+
for item in items:
391+
queue_id = item["queue_id"]
400392

401-
# Remove from queue display
402-
status["queue"] = [qi for qi in status["queue"] if qi["queue_id"] != queue_id]
403-
status["current"] = None
393+
# Skip already completed
394+
if queue_id in completed_ids:
395+
continue
396+
397+
# Skip items with no script
398+
script = REPO_ROOT / item["script"]
399+
if not script.exists():
400+
print(f"[runner] Skipping {queue_id} ({item['claim_id']}) — script not found: {item['script']}", flush=True)
401+
# Update queue item status in live status file
402+
for qi in status["queue"]:
403+
if qi["queue_id"] == queue_id:
404+
qi["status"] = "needs_script"
405+
write_status(status, status_path)
406+
continue
407+
408+
# Run it
409+
result = run_experiment(item, status, status_path, calibration)
410+
ran_any = True
411+
412+
# Move to completed
413+
completed_entry = {
414+
"queue_id": queue_id,
415+
"backlog_id": item.get("backlog_id", ""),
416+
"claim_id": item.get("claim_id", ""),
417+
"title": item.get("title", ""),
418+
"description": item.get("description", ""),
419+
"result": result["result"],
420+
"result_summary": result["result_summary"],
421+
"completed_at": result["completed_at"],
422+
"output_file": result.get("output_file", ""),
423+
}
424+
status["completed"].append(completed_entry)
425+
completed_ids.add(queue_id)
426+
427+
# Remove from queue display
428+
status["queue"] = [qi for qi in status["queue"] if qi["queue_id"] != queue_id]
429+
status["current"] = None
404430

431+
write_status(status, status_path)
432+
print(f"[runner] Done: {queue_id}{result['result']}", flush=True)
433+
434+
# Queue pass complete
435+
if not args.loop:
436+
break
437+
438+
# Loop mode: wait, then reload queue for any newly-added items
439+
status["idle"] = True
440+
status["current"] = None
405441
write_status(status, status_path)
406-
print(f"[runner] Done: {queue_id}{result['result']}", flush=True)
442+
if ran_any:
443+
print(f"[runner] Pass complete. Waiting {args.loop_interval}s before re-checking queue…", flush=True)
444+
else:
445+
print(f"[runner] No new items. Waiting {args.loop_interval}s…", flush=True)
446+
447+
time.sleep(args.loop_interval)
448+
449+
# Reload queue — picks up any new items added while we slept
450+
queue_data = load_queue()
451+
calibration = queue_data.get("calibration", {})
452+
items = queue_data.get("items", [])
453+
454+
new_pending = [i for i in items if i["queue_id"] not in completed_ids]
455+
if new_pending:
456+
print(f"[runner] Found {len(new_pending)} new item(s): "
457+
f"{[i['queue_id'] for i in new_pending]}", flush=True)
458+
# Rebuild queue display in status
459+
new_queue_display = []
460+
for i in new_pending:
461+
new_queue_display.append({
462+
"queue_id": i["queue_id"],
463+
"backlog_id": i.get("backlog_id", ""),
464+
"claim_id": i.get("claim_id", ""),
465+
"title": i.get("title", ""),
466+
"description": i.get("description", ""),
467+
"estimated_minutes": round(estimate_minutes(i, calibration), 1),
468+
"status": "pending",
469+
"status_reason": i.get("status_reason", ""),
470+
})
471+
status["queue"] = new_queue_display
472+
status["idle"] = False
473+
write_status(status, status_path)
407474

408-
# All done
475+
# All done (non-loop path)
409476
status["idle"] = True
410477
status["current"] = None
411478
status["runner_pid"] = None

0 commit comments

Comments
 (0)