Halt after max_consecutive_errored_rollouts#2835
Open
rasdani wants to merge 1 commit into
Open
Conversation
rasdani
force-pushed
the
halt-consecutive-errored-rollouts
branch
from
a668787 to
749b193
Compare
rasdani
changed the title
max_consecutive_errored_rollouts
| max_off_policy_steps: int = Field(8, ge=0) | ||
| """Maximum policies allowed to generate a single rollout. Rollouts generated more than ``max_off_policy_steps`` ahead of training are discarded. Higher values yield better throughput at the cost of off-policy noise.""" | ||
|
|
||
| max_consecutive_errored_rollouts: int | None = Field(10, ge=1) |
Member
There was a problem hiding this comment.
i think 10 is a bit rough here?
Member
There was a problem hiding this comment.
scared of cancelling runs on intermittent errors
Contributor
Author
There was a problem hiding this comment.
what do you suggest?
Contributor
Author
|
@codex remove all tests this PR added |
|
Summary
Testing
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add
orchestrator.max_consecutive_errored_rolloutswith default10andNonedisable support.Stop the training after a configurable number of rollouts failed in a row.
Prevents spamming sandboxes on a persistent sandbox service outage.
Validation
uv run pytest tests/unit/orchestrator/test_orchestrator_error_guard.pyuv run pytest tests/unit/test_configs.py -k max_consecutive_errored_rolloutsuv run ruff check docs/training.md packages/prime-rl-configs/src/prime_rl/configs/orchestrator.py src/prime_rl/orchestrator/orchestrator.py tests/unit/test_configs.py tests/unit/orchestrator/test_orchestrator_error_guard.pygit diff --checkKnown Non-Scope Failure
uv run pytest tests/unit/test_configs.pycurrently fails attest_shared_output_dir_propagates_through_cli; that existing CLI--output-dirpropagation issue is intentionally not fixed in this PR.Note
Low Risk
Localized orchestrator safety rail with default-on threshold and disable via None; no auth or training math changes.
Overview
Adds
orchestrator.max_consecutive_errored_rollouts(default 10,Noneto disable) so long runs fail fast when env rollouts keep dying after verifiers retries are exhausted.The orchestrator counts consecutive terminal errored train and eval rollouts in the main loop before
TrainSink/EvalSinkhandling; successful rollouts reset the streak, and syntheticCancelledmarkers from off-policy cleanup are ignored. Hitting the limit raisesRuntimeErrorwith a warning log per error. Docs and config validation tests cover the new knob; unit tests exercise streak logic and main-loop ordering.Reviewed by Cursor Bugbot for commit 749b193. Bugbot is set up for automated code reviews on this repo. Configure here.