Merge pull request #181 from partach/claude/expand-felicity-card-B7dl4

Claude/expand felicity card b7dl4

Author: partach

CLAUDE.md

@@ -14,6 +14,42 @@ memory or assumptions from training — the facts and status are HERE. If
 something you intend to do contradicts this document, stop and reconcile it
 first.
 
+### The Documentation Set — keep ALL of it current (do not let it float)
+
+There are TWO authoritative docs and they must BOTH be read and kept in sync:
+
+| Doc | Role | Update when |
+|---|---|---|
+| **CLAUDE.md** (this file) | Architecture, status, every fix's rationale, known issues, settings matrix | every change |
+| **docs/EMS_LOGIC.md** | The behavioural SPECIFICATION — what the EMS should do per mode/knob, traced from inputs to action | every change that alters behaviour |
+
+A past session updated only CLAUDE.md and let `docs/EMS_LOGIC.md` go stale —
+that is the "floating basis" failure.  **Before concluding any work, update
+BOTH** (CLAUDE.md for the what/why/status, EMS_LOGIC.md for the behavioural
+contract).  `EMS.md` and the root `EMS_LOGIC.md` are older analysis docs — do
+not treat them as current.
+
+### Validate behaviour with the simulator — don't assert from memory
+
+`tools/ems_simulator.py` runs `ems.calculate_schedule` on a library of named
+scenarios (`tools/scenarios.py`) for BOTH engines, checks expectations, and
+renders charts.  When you change scheduling behaviour: add/adjust a scenario
+that encodes the intended outcome, run `python tools/ems_simulator.py`, and
+confirm it's green before claiming a fix works.  Reproduce customer reports as
+scenarios so they become permanent, readable regression tests.
+
+### Engine default = GREEDY (decision, June 2026)
+
+`scheduler_engine` defaults to **greedy** (multi-month track record, no solver
+dependency).  **MILP is opt-in.**  A prior session flipped the default to MILP;
+it was reverted after fact-checking showed the recent customer-reported bugs
+were MILP-specific or shared-reserve (not greedy scheduling), plus MILP carries
+a pulp/CBC dependency and was not yet validated for determinism.  Do NOT flip
+the default back to MILP without (a) the simulator harness showing MILP is
+deterministic and correct across all knob scenarios, and (b) explicit user
+agreement.  Greedy's two-day reconstruction is less powerful on cross-day
+arbitrage, but it is the proven, dependency-free path.
+
 ### The One Rule That Keeps Breaking — Single Point of Truth
 
 **`ems.py` is the SINGLE SOURCE OF TRUTH for all scheduling logic. It must

custom_components/ha_felicity/__init__.py

@@ -216,7 +216,7 @@ async def async_setup_entry(hass: HomeAssistant, entry: ConfigEntry) -> bool:
         "flexible_load_1_voltage": 230,
         "flexible_load_1_default_current": 16,
         "ev_charge_strategy": "smart",
-        "scheduler_engine": "milp",
+        "scheduler_engine": "greedy",
         "flexible_load_2_enabled": "off",
         "flexible_load_2_name": "",
         "flexible_load_2_switch_entity": "",
@@ -233,17 +233,17 @@ async def async_setup_entry(hass: HomeAssistant, entry: ConfigEntry) -> bool:
         if key not in updated_options:
             updated_options[key] = default
 
-    # One-time default-engine bump: MILP is now the default (it's the robust,
-    # joint 2-day optimiser; greedy's two-day reconstruction is fragile).  Most
-    # existing installs have "greedy" only because that was the OLD default —
-    # not a deliberate choice.  Bump them once.  This is SAFE: when CBC/pulp is
-    # unavailable the MILP auto-disables to greedy (see milp._MILP_DISABLED), so
-    # no install can break.  The marker means we only bump once, so a user who
-    # deliberately re-selects greedy afterwards is respected.
-    if (updated_options.get("scheduler_engine") == "greedy"
-            and not updated_options.get("_scheduler_default_milp_done")):
-        updated_options["scheduler_engine"] = "milp"
-        updated_options["_scheduler_default_milp_done"] = True
+    # Engine default = GREEDY (reverted).  Greedy has a multi-month track
+    # record and no solver dependency; MILP is opt-in until the day-simulator
+    # harness validates it for determinism across all knobs.  Undo the earlier
+    # auto-bump for installs that were AUTO-switched to MILP (marker set) —
+    # surgical and once, so a user who DELIBERATELY selected MILP (no marker)
+    # is never touched.
+    if (updated_options.get("_scheduler_default_milp_done")
+            and not updated_options.get("_scheduler_milp_reverted")):
+        if updated_options.get("scheduler_engine") == "milp":
+            updated_options["scheduler_engine"] = "greedy"
+        updated_options["_scheduler_milp_reverted"] = True
 
     if updated_options != options:
         hass.config_entries.async_update_entry(entry, options=updated_options)

custom_components/ha_felicity/config_flow.py

@@ -108,7 +108,7 @@ def _get_default_options(self) -> dict:
             "flexible_load_1_voltage": 230,
             "flexible_load_1_default_current": 16,
             "ev_charge_strategy": "smart",
-            "scheduler_engine": "milp",
+            "scheduler_engine": "greedy",
             "flexible_load_2_enabled": "off",
             "flexible_load_2_name": "",
             "flexible_load_2_switch_entity": "",

custom_components/ha_felicity/coordinator.py

@@ -679,6 +679,71 @@ def _calculate_net_pv_surplus(self, remaining_slots: list[tuple[int, float]],
             previous_pv_confidence=self._last_pv_confidence,
         )
 
+    def _build_manual_schedule(self) -> None:
+        """Build `scheduled_slots` from the price threshold (manual mode).
+
+        Manual price mode is a simple rule — NOT the optimizer:
+          - from_grid / both: charge every remaining slot whose price is
+            BELOW the threshold.
+          - to_grid / both:   sell every remaining slot whose price is
+            ABOVE the threshold.
+        This keeps the card's displayed schedule consistent with what the
+        manual-mode `_determine_energy_state` actually executes, and ensures
+        we never show a charge slot above the threshold (the optimizer's plan
+        is irrelevant in manual mode).  Past slots are left unscheduled.
+        """
+        opts = self.config_entry.options
+        grid_mode = opts.get("grid_mode", "off")
+        prices = self.slot_prices_today
+        if grid_mode == "off" or not prices or self.price_threshold is None:
+            self.scheduled_slots = {}
+            self._tomorrow_scheduled_slots = {}
+            self.schedule_status = "manual"
+            self.schedule_reason = "Manual mode — follows price threshold"
+            return
+
+        current_slot = self._current_slot_index() or 0
+        allow_charge = grid_mode in ("from_grid", "both")
+        allow_sell = grid_mode in ("to_grid", "both")
+        threshold = self.price_threshold
+
+        sched: dict[int, str] = {}
+        for i in range(current_slot, len(prices)):
+            price = prices[i]
+            if price is None:
+                continue
+            if allow_charge and price < threshold:
+                sched[i] = "charge"
+            elif allow_sell and price > threshold:
+                sched[i] = "discharge"
+
+        self.scheduled_slots = sched
+        # Tomorrow: same threshold rule across all of tomorrow's slots.
+        tmr: dict[int, str] = {}
+        if self.slot_prices_tomorrow:
+            for i, price in enumerate(self.slot_prices_tomorrow):
+                if price is None:
+                    continue
+                if allow_charge and price < threshold:
+                    tmr[i] = "charge"
+                elif allow_sell and price > threshold:
+                    tmr[i] = "discharge"
+        self._tomorrow_scheduled_slots = tmr
+
+        # Clear the optimizer's SOC trajectory so the card doesn't draw a
+        # stale auto-mode line; it falls back to its own client-side
+        # trajectory built from the threshold schedule above.
+        self._backend_soc_trajectory = []
+        self._backend_soc_trajectory_tomorrow = []
+
+        n_charge = sum(1 for v in sched.values() if v == "charge")
+        n_sell = sum(1 for v in sched.values() if v == "discharge")
+        self.schedule_status = "manual"
+        self.schedule_reason = (
+            f"Manual mode: {n_charge} charge / {n_sell} sell slot(s) "
+            f"vs threshold {threshold:.3f}"
+        )
+
     async def _calculate_schedule(self, battery_soc: float | None) -> None:
         """Calculate optimal charge/discharge schedule.
 
@@ -759,7 +824,7 @@ async def _calculate_schedule(self, battery_soc: float | None) -> None:
             )).lower() in ("on", "true", "1"),
             flexible_loads=self._build_flex_load_configs(),
             ev_charge_strategy=str(opts.get("ev_charge_strategy", "smart")),
-            scheduler_engine=str(opts.get("scheduler_engine", "milp")),
+            scheduler_engine=str(opts.get("scheduler_engine", "greedy")),
         )
 
         # What did the previous schedule predict the SOC would be at this slot?
@@ -849,7 +914,7 @@ async def _calculate_schedule(self, battery_soc: float | None) -> None:
             json.dumps(self.slot_overrides, sort_keys=True) if self.slot_overrides else "",
             safe_power_kw,
             opts.get("ev_charge_strategy", "smart"),
-            opts.get("scheduler_engine", "milp"),
+            opts.get("scheduler_engine", "greedy"),
         ))
         if (input_hash == self._last_schedule_input_hash
                 and current_slot_idx == self._last_schedule_slot_idx):
@@ -2471,6 +2536,15 @@ def get_attr(names):
                                 await self._calculate_schedule(battery_soc)
                                 # Schedule may have updated self.price_threshold
                                 new_data["price_threshold"] = self.price_threshold
+                            else:
+                                # Manual mode: the displayed schedule must follow
+                                # the PRICE THRESHOLD, not a stale optimizer plan.
+                                # _calculate_schedule only runs in auto, so
+                                # scheduled_slots would otherwise keep showing the
+                                # last MILP/greedy plan — including charge slots
+                                # ABOVE the threshold, which manual mode must
+                                # never do.  Rebuild it from the threshold here.
+                                self._build_manual_schedule()
 
                             # Always calculate available info (visible in both modes)
                             self._calculate_available_info(battery_soc)

custom_components/ha_felicity/ems.py

@@ -117,7 +117,7 @@ class EMSConfig:
     # "milp" (the solver in milp.py).  When "milp", calculate_schedule tries
     # the MILP first and silently falls back to greedy on any failure
     # (pulp missing, infeasible, timeout).
-    scheduler_engine: str = "milp"
+    scheduler_engine: str = "greedy"
     # NOTE: battery State of Health (SOH) is applied by the coordinator
     # before constructing this config — it scales battery_capacity_kwh
     # by the SOH factor.  ems.py treats the capacity as already-effective.
@@ -1875,11 +1875,22 @@ def calculate_schedule(config: EMSConfig, state: EMSState) -> ScheduleResult:
     if (not state.pv_hourly_kwh
             and forecast_total and forecast_total > 0
             and forecast_remaining and forecast_remaining > 0):
+        # Preserve/synthesize TOMORROW's hourly PV too.  The old rebuild
+        # didn't pass pv_hourly_kwh_tomorrow at all → it was silently dropped
+        # to None, so when today's hourly was absent (forecast gives daily
+        # totals only) the MILP saw tomorrow PV = 0 and planned the whole next
+        # day with NO sun: the SOC trajectory declined all day and it barely
+        # charged (real customer report).  Mirror the today synthesis.
+        pv_hourly_kwh_tomorrow = state.pv_hourly_kwh_tomorrow
+        if (not pv_hourly_kwh_tomorrow
+                and state.pv_forecast_tomorrow and state.pv_forecast_tomorrow > 0):
+            pv_hourly_kwh_tomorrow = _synthesize_pv_hourly(state.pv_forecast_tomorrow)
         state = EMSState(
             battery_soc_pct=state.battery_soc_pct,
             slot_prices_today=state.slot_prices_today,
             slot_prices_tomorrow=state.slot_prices_tomorrow,
             pv_hourly_kwh=_synthesize_pv_hourly(forecast_total),
+            pv_hourly_kwh_tomorrow=pv_hourly_kwh_tomorrow,
             pv_forecast_remaining=forecast_remaining,
             pv_forecast_today=forecast_total,
             pv_forecast_tomorrow=state.pv_forecast_tomorrow,

custom_components/ha_felicity/select.py

@@ -120,7 +120,7 @@ async def async_setup_entry(
             coordinator=coordinator,
             entry=entry,
             option_key="scheduler_engine",
-            select_options=["milp", "greedy"],
+            select_options=["greedy", "milp"],
             name="Scheduler Engine",
             icon="mdi:function-variant",
             entity_category=EntityCategory.CONFIG,

docs/EMS_LOGIC.md

@@ -2,6 +2,47 @@
 
 This document traces the complete decision path of the Felicity EMS, from raw inputs to inverter action, for each mode.
 
+> **This is the behavioural SPECIFICATION.** It must be kept in sync with the
+> code on every behaviour change (alongside CLAUDE.md). The body below predates
+> June 2026; the **Recent behavioural decisions** section is the current
+> authority where it conflicts with older text. Validate with
+> `tools/ems_simulator.py`.
+
+---
+
+## Recent behavioural decisions (June–July 2026) — current authority
+
+These refine/override the older sections below. Each is pinned by a simulator
+scenario and/or `tests/test_ems.py`.
+
+1. **Engine default = greedy; MILP opt-in.** Greedy has a multi-month track
+   record and no solver dependency. MILP is the joint 2-day optimiser but is
+   validated per-scenario before it can become default again.
+2. **Reserve is time-aware (from_grid + MILP all modes).** Past sunset the
+   overnight reserve covers only the REMAINING hours to sunrise, not the full
+   night — so a high-consumption house isn't forced to charge at peak evening
+   prices to "maintain" a full-night reserve the battery should be discharging.
+3. **Self-consumption boost dropped at night.** The ×1.25 boost (hold extra PV
+   energy for self-use) is a daytime concept; at night the reserve is bare
+   survival so the battery rides down and refills from tomorrow's PV.
+4. **Overnight need is profile-aware.** The reserve SUMS the 7-day *hourly*
+   consumption profile over the night hours (not flat daily/24). Critical for
+   daytime-heavy loads (EVs): high daily average but low night use → small
+   reserve → no phantom evening charging. Falls back to flat with no profile.
+   The hourly profile is resolved via the entity registry (exact unique_id),
+   not a guessed entity_id.
+5. **MILP is provably feasible.** Reserve constraints are soft (shortfall slack
+   + penalty) and the SOC dynamics carry a per-slot grid-passthrough slack, so
+   the LP never returns Infeasible (which used to drop ~24% of runs to greedy).
+6. **Manual price mode is a threshold rule, not the optimizer.** `scheduled_slots`
+   in manual mode is rebuilt from the threshold (from_grid/both charge BELOW it,
+   to_grid/both sell ABOVE it) — never the stale optimizer plan.
+7. **No charge deferral in the coordinator.** Every scheduled charge slot
+   executes; the cheapest-slot decision lives entirely in `ems.py`.
+8. **Manual slot picks are grid-mode-aware.** from_grid → any picked slot is a
+   charge slot (even above threshold); to_grid → any is a sell slot; both →
+   threshold decides.
+
 ---
 
 ## 1. User Configuration: The Settings That Shape Every Decision

tests/test_ems.py

@@ -5058,10 +5058,11 @@ def test_sell_coverage_adds_today_charge_slots(self):
 class TestMILPScheduler:
     """Tests for the optional MILP scheduler (milp.py)."""
 
-    def test_default_engine_is_milp(self):
-        """MILP is now the default engine (robust joint 2-day optimiser)."""
+    def test_default_engine_is_greedy(self):
+        """Greedy is the default engine (proven track record, no solver dep).
+        MILP is opt-in until the day-simulator harness validates it."""
         config = EMSConfig(grid_mode="from_grid", battery_capacity_kwh=10)
-        assert config.scheduler_engine == "milp"
+        assert config.scheduler_engine == "greedy"
 
     def test_milp_never_infeasible_extreme_drain(self):
         """MILP must stay feasible (not fall back) even when consumption would

tools/README.md

@@ -0,0 +1,72 @@
+# EMS day-simulator / scenario harness
+
+A standalone way to **see and confirm what the scheduling algorithm actually
+does** — for both engines (greedy and MILP) — without Home Assistant.  It runs
+`ems.calculate_schedule` on a library of realistic scenarios, prints a report,
+checks per-scenario expectations, and (optionally) renders a chart per scenario.
+
+## Run it (Windows / macOS / Linux)
+
+```bat
+cd ha_felicity
+python -m pip install pulp matplotlib
+python tools\ems_simulator.py
+```
+
+- `pulp` enables the **MILP** engine (without it, only greedy runs).
+- `matplotlib` enables the **charts** (without it you still get the full text report).
+
+Charts are written to `tools\sim_output\<scenario>.png` — one image per
+scenario with two stacked panels (greedy on top, MILP below): price bars
+coloured **green = charge / orange = sell / grey = idle**, with the projected
+**SOC %** line and the **reserve** line overlaid.
+
+### Useful flags
+
+```bat
+python tools\ems_simulator.py --name self_suff_daytime_ev   :: one scenario
+python tools\ems_simulator.py --engine greedy               :: one engine
+python tools\ems_simulator.py --no-plot                     :: text only
+```
+
+The process exits **0** when every scenario expectation passes, **1** if any
+fail — so it can gate a release.
+
+## What's covered
+
+`tools/scenarios.py` holds the scenario library.  Each scenario exercises
+specific knobs and asserts the intended outcome.  The current set covers:
+
+| Scenario | Knobs exercised | What it pins |
+|---|---|---|
+| `save_money_cheap_night` | from_grid, cost, reserve_target_pct | charges only the cheapest slots |
+| `self_suff_daytime_ev` | self_consumption, hourly profile, evening | **customer case**: no peak evening charge at 80% SOC |
+| `self_suff_flat_low_soc` | self_consumption, low SOC midday | charges to cover the deficit |
+| `trader_arbitrage` | both, cost, wide spread | buys cheap, sells above buy price |
+| `to_grid_sell_surplus` | to_grid, high SOC, PV | never charges from grid |
+| `negative_prices_charge_to_full` | charge_to_full_on_negative_price | greedy grabs all p<0; MILP charges p<0 |
+| `low_soc_urgent_recovery` | from_grid, SOC < min | forces immediate charging |
+| `tomorrow_pv_daily_only` | daily-only forecast, two-day | tomorrow's PV is synthesised (not zero) |
+| `longevity_cycle_cost` | both, longevity | wear floor suppresses marginal trades |
+| `arbitrage_delta_gate` | both, arbitrage_price_delta | no sells below the required spread |
+
+## Add your own
+
+Append a dict to `SCENARIOS` in `tools/scenarios.py`:
+
+```python
+{
+  "name": "my_case",
+  "desc": "what it checks",
+  "config": dict(grid_mode="from_grid", optimization_priority="self_consumption",
+                 battery_capacity_kwh=48.0, ...all the knobs...),
+  "state":  dict(battery_soc_pct=80.0, slot_prices_today=[...],
+                 pv_hourly_kwh={...}, consumption_hourly_kwh={...},
+                 current_hour=22, current_minute=30),
+  "expect": lambda r, s: (r["charge_slots"] == [], "should not charge"),
+}
+```
+
+Reproduce a customer screenshot by transcribing its prices / SOC / time / knobs
+into a scenario — then the expected behaviour becomes a permanent, runnable
+regression test that anyone can read.