feat: add orchestrated backtest pipeline (sweep -> walk-forward -> monte carlo)

[michaelchu]

When sweep_params are provided, the backtest tool now runs a full analysis
pipeline (pipeline=true): sweep -> significance gate ->
walk-forward validation -> OOS data gate -> monte carlo risk simulation.

Each stage reports a StageStatus (completed/skipped/failed) with reasons,
designed for frontend rendering of pass/fail gate cards. Users can opt out
with pipeline=false (default) to get just the sweep result.

Key changes:

• Extract execute_from_returns() from monte_carlo.rs for in-process reuse
• Add PipelineResponse and StageInfo types for frontend-renderable stages
• Add pipeline.rs orchestrator with significance and OOS data gates
• Add Pipeline variant to BacktestToolResponse enum
• Add MIN_RETURNS_FOR_BOOTSTRAP constant (30) to constants.rs

https://claude.ai/code/session_01DEHwjSk7Y38DhefWeGZCLu

[Copilot]

Pull request overview

Adds an orchestrated “backtest pipeline” mode to the backtest tool so that, when sweeps are requested, the server can automatically chain follow-on analysis stages (walk-forward + Monte Carlo) and report per-stage statuses for frontend gate rendering.

Changes:

• Introduces PipelineResponse / StageInfo / StageStatus response types and wires them into BacktestToolResponse as a new pipeline variant.
• Adds a new src/tools/pipeline.rs orchestrator that runs sweep → significance gate → walk-forward → OOS data gate → Monte Carlo.
• Refactors Monte Carlo to support execute_from_returns() for reuse by the pipeline; adds MIN_RETURNS_FOR_BOOTSTRAP constant.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
src/tools/response_types/pipeline.rs	New pipeline response schema for stage/gate rendering.
src/tools/response_types/mod.rs	Exposes the new pipeline response types.
src/tools/pipeline.rs	Implements sweep-following orchestration and gating logic.
src/tools/monte_carlo.rs	Extracts execute_from_returns() so Monte Carlo can run on derived returns.
src/tools/mod.rs	Registers the new pipeline tool module.
src/tools/backtest.rs	Adds pipeline param (default true for sweeps) and returns a new pipeline response variant.
src/server/mod.rs	Updates tool docs to mention the new pipeline behavior.
src/constants.rs	Adds MIN_RETURNS_FOR_BOOTSTRAP constant for Monte Carlo suitability gating.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

[michaelchu]

@copilot apply changes based on the comments in this thread

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

[Copilot]

stages["sweep"].duration_ms is populated from sweep_response.execution_time_ms, but when num_permutations > 0 the permutation gate work happens after the sweep completes (in execute_sweep_raw via spawn_blocking) and does not update execution_time_ms. This can materially under-report the sweep stage duration in pipeline UI when permutations are enabled. Consider returning the total sweep+permutation duration from execute_sweep_raw (or measuring elapsed time around sweep+permutation) and using that for the sweep stage duration, or splitting permutation testing into its own stage with its own duration.

Suggested change

	// Stage 1: Sweep (already completed)
	stages.push(StageInfo {
	name: "sweep".to_string(),
	status: StageStatus::Completed,
	reason: None,
	duration_ms: sweep_response.execution_time_ms,
	let num_permutations = base_params
	.get("num_permutations")
	.and_then(Value::as_u64)
	.unwrap_or(0);
	let sweep_response_value = serde_json::to_value(&sweep_response).ok();
	let total_sweep_duration_ms = sweep_response_value.as_ref().and_then(|value| {
	value
	.get("total_execution_time_ms")
	.and_then(Value::as_u64)
	.and_then(|duration_ms| duration_ms.try_into().ok())
	.or_else(|| {
	value
	.get("permutation_execution_time_ms")
	.and_then(Value::as_u64)
	.and_then(|duration_ms| duration_ms.try_into().ok())
	.map(|permutation_duration_ms| {
	sweep_response
	.execution_time_ms
	.saturating_add(permutation_duration_ms)
	})
	})
	});
	let sweep_stage_duration_ms =
	total_sweep_duration_ms.unwrap_or(sweep_response.execution_time_ms);
	let sweep_stage_reason = if num_permutations > 0 && total_sweep_duration_ms.is_none() {
	Some(
	"Permutation testing was enabled, but the sweep response did not include total sweep+permutation timing; displaying the reported sweep execution time only."
	.to_string(),
	)
	} else {
	None
	};
	// Stage 1: Sweep (already completed)
	stages.push(StageInfo {
	name: "sweep".to_string(),
	status: StageStatus::Completed,
	reason: sweep_stage_reason,
	duration_ms: sweep_stage_duration_ms,