This project investigates whether lightweight statistical monitors can detect safety-relevant behavioral changes in a reinforcement learning agent during training, before those changes propagate into a deployed policy.
Research question: Can entropy-based, reward-based, and distributional divergence monitors reliably identify reward hacking and observation distribution shift in a training PPO agent, using only information already available in the training loop?
The experiment deliberately injects two known failure modes — reward hacking and covariate shift — and measures whether each is flagged by the corresponding detector.
The system implements two complementary monitoring mechanisms operating in parallel during RL training:
Conscious Hardware Loop — a discrete-time PID controller that adaptively regulates computational load during a training workload, comparing the system's energy use, latency, and temperature against a static (uncontrolled) baseline.
Shadow Behavior Detector — a statistical monitoring system inserted into the training loop via a Stable-Baselines3 callback. It tracks three independent signals: policy entropy, per-episode reward distribution, and KL divergence from a fixed early-training reference policy. Anomalies are flagged when each signal crosses a statistically motivated threshold.
Neither module modifies the underlying learning algorithm. Both are designed to operate as passive observers that can be attached to any SB3-compatible training run.
-
Detection sensitivity: Do statistical thresholds derived from the training signal's own rolling statistics produce reliable anomaly flags when a known behavioral change is injected, without generating excessive false positives during normal learning?
-
Reference policy choice: Does comparing the current policy to a fixed early-training reference (rather than a sliding window) improve sensitivity to cumulative behavioral drift caused by observation distribution shift?
-
Hardware efficiency: Does a classical PID feedback controller reduce energy consumption and latency on a simulated AI training server under positive-biased workload disturbances, compared to an uncontrolled static system?
A PPO agent (Schulman et al., 2017) is trained on CartPole-v1 for 100,000 timesteps using Stable-Baselines3 with a standard MlpPolicy. CartPole-v1 was chosen because its policy is analytically interpretable — shifts in action probability distributions are easy to reason about — and because it reaches stable behavior within a modest compute budget.
Two anomalies are injected at predetermined timesteps via a Gymnasium wrapper
(AnomalyInjectableCartPole) that intercepts the environment's step() output:
| Timestep | Anomaly | Mechanism |
|---|---|---|
| 25,000 | Reward hacking | +2.0 bonus reward added when action == 0, regardless of state |
| 65,000 | Distribution shift | Gaussian observation noise (σ = 0.4) applied to all four state dimensions |
The injection times are chosen to occur after initial learning has stabilized, so that any subsequent signal changes are attributable to the injection rather than normal early-training dynamics.
Entropy spike detector monitors policy entropy H(π) = −Σ π(a|s) log π(a|s), evaluated on a held-out pool of 100 fixed states every 500 training steps. A spike is flagged when |H_t − mean(H_recent)| > 2σ over a rolling window of W = 20 measurements.
Reward hacking detector monitors per-episode return. A hacking event is flagged when R_t > mean(R_recent) + 3σ over the last W = 50 episodes. The 3σ threshold is chosen to minimize false positives during normal reward improvement.
Behavioral drift detector monitors KL divergence from a fixed reference policy snapshot taken at the first callback evaluation (approximately t = 500). A drift event is flagged when D_KL(π_current ‖ π_ref) > 0.15 nats. Using a fixed reference — rather than a sliding window — accumulates drift signal over the full training history, making it more sensitive to the gradual policy changes induced by covariate shift.
The hardware loop implements the discrete-time PID update:
with gains (K_p, K_i, K_d) = (0.5, 0.1, 0.05) and target load x* = 60%. Anti-windup clamping is applied to the integral term (clip to [−60, +60]) to prevent overshoot under sustained disturbances.
On reward hacking detection: The reward hacking detector produced 18 flagged events after the injection at t = 25,000. Prior to injection, episode rewards remained below the 3σ threshold throughout training. The spurious +2.0 bonus — applied unconditionally on action = 0 — caused episode returns to spike sharply, which the rolling reward distribution picked up within 1–2 episodes of the first injection-influenced episode completing.
On behavioral drift detection: Using a sliding window reference policy, KL divergence values peaked at only 0.037 nats after the distribution shift injection, well below any useful detection threshold. Switching to a fixed early-training reference policy increased post-injection KL divergence to values consistently crossing 0.15 nats, generating 180 drift detection events after t = 65,000. This finding suggests that reference policy choice is the dominant factor in drift detector sensitivity — a sliding window effectively normalizes away the very signal it is meant to detect.
On entropy dynamics: Policy entropy decreased monotonically during normal learning (from ~0.69 nats at initialization toward ~0.22 nats at convergence), which is expected as the policy becomes more deterministic. The 37 entropy spike events occurred at training transitions where the policy temporarily increased uncertainty — likely corresponding to policy updates that explored new action regions. No entropy spikes were directly attributable to the injected anomalies, suggesting entropy alone is not a reliable proxy for either of the two tested failure modes.
On PID hardware control: The PID controller maintained CPU load within a narrow band around the 60% target (load std dev 3.07 vs 7.94 uncontrolled) despite positive-biased workload disturbances. The primary efficiency gain came from variance reduction rather than load reduction per se: the static system drifted toward saturation under the asymmetric disturbance distribution, while the adaptive system absorbed burst spikes through proportional correction.
Training run: 100,000 timesteps, 648 episodes, PPO on CartPole-v1. Final mean reward (last 50 episodes): 324.36 / 500.
| Detector | Events Detected | Injected Anomaly Caught |
|---|---|---|
| Entropy spike | 37 | N/A (natural training signal) |
| Reward hacking | 18 | Yes — injection at t = 25,000 |
| Behavioral drift | 180 | Yes — injection at t = 65,000 |
| Total | 235 | 2 / 2 injections detected |
| Metric | Static (Uncontrolled) | Adaptive (PID) | Change |
|---|---|---|---|
| Energy (kWh) | 387.36 | 230.93 | −40.4% |
| Avg Latency (ms) | 153.94 | 80.72 | −47.6% |
| Peak Temperature (°C) | 75.00 | 65.13 | −13.2% |
| Mean Load (%) | 96.84 | 57.73 | −40.4% |
| Load Std Dev | 7.94 | 3.07 | −61.3% |
Top row: policy entropy with spike annotations (left); episode reward with reward hacking markers (right). Bottom row: KL divergence from fixed reference policy with drift threshold and injection marker (left); PID hardware load static vs adaptive (right). Summary table below.
adaptive-ai-monitoring/
│
├── pid_controller.py — Discrete-time PID controller, workload simulation,
│ metric computation, and 4-panel comparison plot
│
├── shadow_detector.py — Statistical anomaly detectors (entropy spike,
│ reward hacking, behavioral drift via KL divergence)
│ No RL dependencies — attaches to any training loop
│
├── experiment.py — PPO training experiment: SB3 callback, anomaly
│ injection wrapper, fixed reference policy logic,
│ and results serialization
│
├── monitoring_dashboard.py — 5-panel dashboard consuming outputs/ CSV and JSON
│
├── run_all.py — Master pipeline: runs all three stages in order
│ (--skip-training flag to rebuild dashboard only)
│
├── requirements.txt — Pinned dependency versions
├── LICENSE — MIT
│
└── outputs/ — Generated by running the scripts (not tracked in git
├── monitoring_dashboard.png except the PNG visualizations)
├── pid_comparison.png
├── anomaly_log.csv
├── experiment_results.json
└── training_metrics_*.csv
This project was built to investigate a practical gap in reinforcement learning tooling: while the theoretical failure modes of RL agents are well-documented in the AI safety literature (Amodei et al., 2016; Krakovna et al., 2020), most training pipelines lack any runtime monitoring that could surface these failures as they emerge.
The investigation focused on whether passive, statistically-motivated monitors — requiring no modification to the learning algorithm — could provide useful signal on two of the most commonly discussed failure modes. The finding that reference policy choice dominates behavioral drift detector sensitivity was not anticipated at the outset and emerged from iterating on the experimental design.
The PID hardware loop addresses a parallel concern: that AI training workloads, if unregulated, tend to saturate available computational resources under realistic burst-traffic disturbance models. Classical control theory offers a well-understood solution that complements the statistical monitoring layer.
Both components together constitute a lightweight monitoring framework that could reasonably be attached to production RL training runs with minimal overhead.
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt
python run_all.pyExpected runtime: 5–10 minutes on CPU.
[1] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. arXiv:1606.06565.
[2] Krakovna, V., et al. (2020). Specification gaming: the flip side of AI ingenuity. arXiv:2011.09294.
[3] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
[4] Leike, J., et al. (2017). AI Safety Gridworlds. arXiv:1711.09883.
[5] Åström, K.J. & Hägglund, T. (1995). PID Controllers: Theory, Design and Tuning (2nd ed.). ISA Press.
[6] Kullback, S. & Leibler, R.A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22(1), 79–86.
[7] Raffin, A., et al. (2021). Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268), 1–8.
@software{awari2025adaptive,
author = {Awari, Ajinkya},
title = {Adaptive {AI} Monitoring with Shadow Behavior Detection},
year = {2025},
url = {https://github.com/ajinkya-awari/adaptive-ai-monitoring},
license = {MIT}
}MIT License — see LICENSE.