Merge pull request #186 from partach/claude/expand-felicity-card-B7dl4

Claude/expand felicity card b7dl4

Author: partach

CLAUDE.md

@@ -74,7 +74,7 @@ this rule exists to prevent — don't.
 
 ### Before Concluding Any Work
 
-- Run `python -m pytest tests/test_ems.py` (must stay green; currently 228).
+- Run `python -m pytest tests/test_ems.py` (must stay green; currently 244).
 - If you added a setting, update the **Settings Traceability Matrix** below
   and confirm it is consumed by the algorithm (no "optimized-out" settings).
 - Keep this document in sync with the code. If status changed, update it.
@@ -115,7 +115,7 @@ custom_components/ha_felicity/
     └── ha_felicity_ems.js   # LitElement EMS dashboard card (1671 lines)
 
 tests/
-└── test_ems.py              # 234 tests for the pure EMS algorithm
+└── test_ems.py              # 244 tests for the pure EMS algorithm
 ```
 
 ---
@@ -594,6 +594,38 @@ The validation simulates forward through PV-only overflows (clamping
 soc and continuing) instead of breaking on the first PV-caused
 violation, which lets it detect phantom charges later in the day.
 
+**Re-shop after overflow drop (greedy from_grid, July 2026)
+— `_fill_charge_to_deficit`**: the greedy selector picks the cheapest slots
+cheapest-first.  When the two cheapest slots land *back-to-back* on a small
+battery, charging them consecutively overflows capacity, so SOC validation
+drops one — and the dropped energy was previously never re-allocated, leaving
+the overnight deficit half-covered.  Real symptom: a 20 kWh battery under a
+40 kWh/day load charged only **1 of 2** needed cheap night slots, then rode
+the floor ~9 h earlier than the MILP plan (which placed its second charge
+*later*, after consumption had freed headroom).  After validation,
+`_schedule_from_grid` now re-shops: it walks the remaining slots
+cheapest-first and adds each candidate that (a) is priced **≤ the marginal
+price the selector already committed to**, (b) survives a fresh validation
+against the kept set, and (c) increases delivered charge energy — until the
+deficit is covered.  Two guards keep it cost-first:
+- **Price ceiling** (`max_price` = max price of the selected set): we only
+  re-shop into a slot as cheap as the ones already chosen.  We must NOT reach
+  for a pricier slot to chase the deficit — bridging an overnight need by
+  charging a 0.30 evening peak (while the chosen slots were 0.15, or while
+  cheap 0.05 night grid follows after midnight) costs MORE than drawing the
+  shortfall from the grid live (round-trip loss on a higher price).  Same
+  no-safety-swap economics the two-day selector applies.
+- **PV-aware headroom cap** (`min(deficit, max_battery − current − net_pv)`):
+  when PV surplus will fill the battery, the target shrinks so we don't
+  over-charge.
+Scoped to greedy `from_grid` and skipped for
+`charge_to_full_on_negative_price` (its negative-slot handling is outside the
+deficit model).  No-op for healthy schedules (validation dropped nothing →
+delivered energy already meets the target).  Pinned by
+`TestGreedyReshopAfterOverflow` (re-shops a cheap slot; never an expensive
+one).  This closes most of the greedy/MILP gap on undersized-battery /
+heavy-load days without touching MILP.
+
 ### Arbitrage Price Delta (both mode) — the TRADE TRIGGER
 
 `arbitrage_price_delta` is the explicit minimum buy→sell spread required
@@ -1201,8 +1233,26 @@ Forward-simulates SOC through every slot.  Drops violations:
   where `terminal = avg_price × efficiency`.
 - **Price gates**: `arbitrage_price_delta` → charge/discharge UB = 0 for
   slots outside spread.  `block_export_on_negative_price` → discharge UB = 0.
-- **Post-solve**: LP allocations ranked by energy, capped by battery
-  physics (charge headroom / discharge headroom) to prevent over-scheduling.
+- **Post-solve (cost-ranked discrete extraction)**: the continuous LP is
+  collapsed to discrete full-power slots, capped by battery physics (charge /
+  discharge headroom) to prevent over-scheduling.  The collapse executes the
+  most cost-effective subset of the slots the LP used:
+  - **Discharge: dearest-first.**  When the cap allows fewer sell slots than
+    the LP spread energy across (the LP cycles PV — sell early to clear room,
+    then sell again at the peak, so the early cheap slot and the peak tie on
+    energy), keep the **highest-price** slots.  (Ranking by LP energy used to
+    keep the cheap early slot over the 0.40 peak — `to_grid` bug #3.)
+  - **Charge: cheapest-first, per day.**  Cap each day at the energy the LP
+    allocated to it (preserving the midnight-reserve **today-first** split — a
+    single global cheapest-first cap would let cheap tomorrow slots starve
+    today), then within each day pick the **cheapest** slots.  A **half-slot
+    rule** (only add a slot if the remaining day-target exceeds half its
+    energy) stops a pricier slot being pulled in to cover a tiny remainder —
+    which is how an expensive reserve-top-off slot used to sneak in (MILP
+    charging a 0.30 peak to hit a high self_consumption reserve — bug #2).
+  Pinned by `test_milp_sells_evening_peak_not_cheap_early_slot`,
+  `test_milp_no_peak_charge_to_hit_reserve`, and the `to_grid_sell_surplus` /
+  `self_suff_flat_low_soc` simulator scenarios.
 - Returns `(today_slots, tomorrow_slots)` or `None` → greedy fallback.
 
 ### 6. Post-Processing (both engines)
@@ -1765,7 +1815,7 @@ in the solver (loads as decision variables, not just overlays).
 
 ## Testing
 
-Tests are in `tests/test_ems.py` (234 tests). They import `ems.py` directly (bypassing HA dependencies) and test the pure scheduling functions.
+Tests are in `tests/test_ems.py` (244 tests). They import `ems.py` directly (bypassing HA dependencies) and test the pure scheduling functions.
 
 ```bash
 # Run all tests

custom_components/ha_felicity/ems.py

@@ -960,6 +960,129 @@ def _validate_schedule_soc(
     return charge_slots, discharge_slots
 
 
+def _slot_grid_charge_kwh(
+    slot_idx: int,
+    pv_hourly_kwh: dict[int, float] | None,
+    minutes_per_slot: float,
+    pv_confidence: float,
+    energy_per_slot: float,
+    efficiency: float,
+    inverter_max_power_kw: float,
+    safe_power_kw: float,
+) -> float:
+    """Grid energy (kWh, after charge efficiency) a single charge slot can
+    actually store, accounting for PV sharing the inverter at that hour.
+
+    Mirrors the per-slot charge contribution computed inside
+    `_validate_schedule_soc`, so callers can size how much a candidate slot
+    adds without re-running the full SOC simulation.
+    """
+    if inverter_max_power_kw > 0:
+        hour = int((slot_idx * minutes_per_slot) / 60)
+        pv_kwh = (pv_hourly_kwh or {}).get(hour, 0.0) * pv_confidence
+        grid_kw = min(
+            safe_power_kw or energy_per_slot / (minutes_per_slot / 60.0),
+            max(0.0, inverter_max_power_kw - pv_kwh),
+        )
+        return grid_kw * (minutes_per_slot / 60.0) * efficiency
+    return energy_per_slot * efficiency
+
+
+def _fill_charge_to_deficit(
+    remaining: list[tuple[int, float]],
+    validated_charge: set[int],
+    energy_deficit: float,
+    current_kwh: float,
+    consumption_per_slot: float,
+    pv_hourly_kwh: dict[int, float] | None,
+    minutes_per_slot: float,
+    pv_confidence: float,
+    battery_capacity: float,
+    min_kwh: float,
+    energy_per_slot: float,
+    efficiency: float,
+    inverter_max_power_kw: float,
+    safe_power_kw: float,
+    consumption_hourly_kwh: dict[int, float] | None,
+    max_price: float | None = None,
+) -> set[int]:
+    """Re-shop charge energy dropped by SOC validation into later slots.
+
+    The greedy selector (`select_unified_charge_slots`) picks the cheapest
+    slots cheapest-first.  When the cheapest slots land back-to-back on a
+    small battery, charging them consecutively overflows capacity, so
+    `_validate_schedule_soc` drops one — and the dropped energy is otherwise
+    never re-allocated, leaving the overnight deficit only partly covered
+    (real symptom: a 20 kWh battery under a 40 kWh/day load charged only one
+    of two needed cheap slots, then rode the floor ~9 h earlier than the MILP
+    plan, which placed its second charge *later*, after consumption had freed
+    headroom).
+
+    This walks the remaining slots cheapest-first and adds each candidate that
+    (a) is priced at or below ``max_price`` (the marginal price the selector
+    already committed to), (b) survives a fresh validation against the
+    already-kept set, and (c) increases the delivered charge energy.  Adding a
+    slot *after* prior drain is exactly what lets the second charge fit.
+
+    The ``max_price`` ceiling is the cost-discipline guard: we only re-shop the
+    dropped energy into a slot as cheap as the ones already chosen.  We must
+    NOT reach for a more expensive slot to chase the deficit — if the only
+    feasible later slot is pricier (e.g. a 0.30 evening peak while the chosen
+    slots were 0.15, or while cheap 0.05 night grid follows after midnight),
+    bridging the overnight need by charging that expensive slot costs MORE than
+    just drawing the shortfall from the grid live (round-trip losses on a
+    higher price).  This is the same no-safety-swap economics the two-day
+    selector already applies.  Cheapest-first + the ceiling keep us cost-first;
+    we stop the moment the deficit is covered, so the battery is never charged
+    beyond what overnight survival needs.  In the common case where validation
+    dropped nothing, the delivered energy already meets the deficit and this
+    returns immediately — a no-op for healthy schedules.
+    """
+
+    def delivered(slots: set[int]) -> float:
+        return sum(
+            _slot_grid_charge_kwh(
+                s, pv_hourly_kwh, minutes_per_slot, pv_confidence,
+                energy_per_slot, efficiency, inverter_max_power_kw,
+                safe_power_kw,
+            )
+            for s in slots
+        )
+
+    validated = set(validated_charge)
+    got = delivered(validated)
+    if got >= energy_deficit - 0.01:
+        return validated
+
+    attempted = set(validated)
+    pool = sorted(
+        (
+            (idx, price) for idx, price in remaining
+            if price is not None and idx not in attempted
+            and (max_price is None or price <= max_price + 0.0001)
+        ),
+        key=lambda ip: ip[1],
+    )
+    for idx, _price in pool:
+        if got >= energy_deficit - 0.01:
+            break
+        attempted.add(idx)
+        trial, _ = _validate_schedule_soc(
+            remaining, validated | {idx}, set(),
+            current_kwh, consumption_per_slot, pv_hourly_kwh,
+            minutes_per_slot, pv_confidence, battery_capacity, min_kwh,
+            energy_per_slot, efficiency,
+            inverter_max_power_kw=inverter_max_power_kw,
+            safe_power_kw=safe_power_kw,
+            consumption_hourly_kwh=consumption_hourly_kwh,
+        )
+        new_got = delivered(trial)
+        if new_got > got + 0.01:
+            validated = trial
+            got = new_got
+    return validated
+
+
 def _select_discharges_for_pv_headroom(
     remaining: list[tuple[int, float]],
     current_kwh: float,
@@ -2257,7 +2380,42 @@ def _schedule_from_grid(
         safe_power_kw=config.safe_power_kw,
         keep_all_negative_charges=config.charge_to_full_on_negative_price,
     )
-    selected = [(idx, p) for idx, p in selected if idx in validated_charge]
+
+    # Re-shop any charge energy that validation dropped for overflow into
+    # later slots where prior consumption has freed headroom, so the overnight
+    # deficit is actually covered (see _fill_charge_to_deficit).  Skipped for
+    # charge_to_full_on_negative_price, whose negative-slot handling and
+    # PV-curtailment trade-off are intentionally outside the deficit model.
+    if not config.charge_to_full_on_negative_price:
+        # Bound the re-shop by the SAME PV-aware headroom the selector uses
+        # (max_battery − current − net_pv_surplus).  This is the cost
+        # discriminator: when PV surplus will fill the battery (net_pv > 0),
+        # headroom is small, so we never reach for an expensive evening slot
+        # to chase the deficit — the original cheap slot + solar already
+        # cover it (a small battery at midday under good PV must NOT buy
+        # peak-priced evening grid just to top off; that violates
+        # cost-first).  When PV can't fill the battery (net_pv ≈ 0, heavy
+        # load), headroom is large and the dropped cheap slot is re-shopped.
+        fill_headroom = max(0.0, max_battery_kwh - current_kwh - max(0.0, net_pv))
+        fill_target = min(energy_deficit, fill_headroom)
+        # Marginal price the selector already committed to: never re-shop the
+        # dropped energy into a pricier slot (see _fill_charge_to_deficit).
+        fill_max_price = max((p for _, p in selected if p is not None), default=None)
+        validated_charge = _fill_charge_to_deficit(
+            remaining, validated_charge, fill_target,
+            current_kwh, consumption_per_slot, state.pv_hourly_kwh,
+            minutes_per_slot, pv_confidence, config.battery_capacity_kwh,
+            min_kwh, energy_per_slot, config.efficiency,
+            config.inverter_max_power_kw, config.safe_power_kw,
+            state.consumption_hourly_kwh,
+            max_price=fill_max_price,
+        )
+
+    price_of_remaining = {idx: p for idx, p in remaining}
+    selected = [
+        (idx, price_of_remaining.get(idx, 0.0)) for idx in validated_charge
+    ]
+    selected.sort(key=lambda ip: ip[0])
 
     # discharge_to_make_room_for_negative_price: in from_grid mode we
     # normally don't discharge.  When this opt-in is enabled, schedule

custom_components/ha_felicity/milp.py

@@ -345,9 +345,18 @@ def _add_day(prices: list[float | None], pv_hourly: dict[int, float],
         if dv > slot_discharge_cap * MIN_FRAC:
             discharge_candidates.append((k, h, dv))
 
-    # Sort by LP-allocated energy descending — highest-conviction slots first.
-    charge_candidates.sort(key=lambda x: -x[2])
-    discharge_candidates.sort(key=lambda x: -x[2])
+    # Rank for the capped, discrete, full-power execution.  The LP spreads
+    # energy across more slots than the inverter can run at full power, so we
+    # take a *subset* up to the physical headroom (below).  Among the slots the
+    # LP chose to use, execute the most COST-EFFECTIVE subset: most-expensive
+    # slots to SELL (charge is ranked per-day in the extraction below, to
+    # respect the today/tomorrow split).  Ranking discharge by LP-allocated
+    # energy instead (the old behaviour) could keep an early CHEAP-priced sell
+    # over the evening PEAK when both got similar energy — the LP cycles PV
+    # (sell early to make room, then sell again at the peak), so both tie on
+    # energy and the cheaper one used to win.  Price is the primary key; LP
+    # energy is the tiebreaker (conviction).
+    discharge_candidates.sort(key=lambda x: (-x[1]["price"], -x[2]))
 
     # Compute energy caps from battery physics.  The LP's continuous
     # totals include marginal fractional allocations the inverter can't
@@ -377,16 +386,34 @@ def _add_day(prices: list[float | None], pv_hourly: dict[int, float],
     today_scheduled: dict[int, str] = {}
     tomorrow_scheduled: dict[int, str] = {}
 
-    accum = 0.0
-    for k, h, kw in charge_candidates:
-        if accum >= charge_target_kwh:
-            break
-        slot_energy = h["charge_cap"] or safe_kwh
-        accum += slot_energy * eff  # effective stored energy at full power
-        if h["day"] == "today":
-            today_scheduled[h["slot"]] = "charge"
-        else:
-            tomorrow_scheduled[h["slot"]] = "charge"
+    # --- Charge execution: per day, cheapest-first ---
+    # Collapse the LP's continuous charge plan to discrete full-power slots.
+    # Cap EACH DAY at the effective energy the LP allocated to it, so the
+    # today/tomorrow split the solver chose is preserved: the midnight-reserve
+    # constraint decides how much MUST charge today (self-sufficiency
+    # "today-first") versus deferring to a cheaper tomorrow — a single global
+    # cap + cheapest-first would let cheap tomorrow slots starve today.
+    # WITHIN each day, execute the CHEAPEST slots, and never pull in a pricier
+    # slot just to cover a sub-half-slot remainder — that half-slot rule is
+    # what stops an expensive reserve-top-off slot from sneaking in (#2: a
+    # 0.30 evening slot used to be added to "finish" a reserve the two cheap
+    # 0.15 slots had effectively already met).
+    for day, sched in (("today", today_scheduled),
+                       ("tomorrow", tomorrow_scheduled)):
+        cands = sorted(
+            ((h, cv) for (k, h, cv) in charge_candidates if h["day"] == day),
+            key=lambda x: (x[0]["price"], -x[1]),
+        )
+        target = eff * sum(cv for _, cv in cands)
+        if day == "today":
+            target = min(target, charge_target_kwh)
+        accum = 0.0
+        for h, cv in cands:
+            slot_energy = (h["charge_cap"] or safe_kwh) * eff
+            if target - accum <= slot_energy * 0.5:
+                break  # remaining need < half a slot — don't add a pricier one
+            accum += slot_energy
+            sched[h["slot"]] = "charge"
 
     accum = 0.0
     for k, h, kw in discharge_candidates:

docs/EMS_LOGIC.md

@@ -52,6 +52,42 @@ scenario and/or `tests/test_ems.py`.
 8. **Manual slot picks are grid-mode-aware.** from_grid → any picked slot is a
    charge slot (even above threshold); to_grid → any is a sell slot; both →
    threshold decides.
+9. **Greedy re-shops charge energy dropped for overflow (from_grid).** When SOC
+   validation drops a back-to-back charge slot that overflowed a small battery,
+   `_schedule_from_grid` re-shops the dropped energy into a LATER slot where
+   prior drain freed headroom — but only at a price **≤ the marginal price the
+   selector already committed to**, and bounded by PV-aware headroom. So a
+   heavy-load / undersized-battery day charges the second *cheap* slot (closing
+   the greedy/MILP gap — both reach the same planned kWh), while a PV-fed small
+   battery never buys an expensive evening peak to chase the deficit
+   (cost-first / no-safety-swap: drawing the shortfall live from cheap night
+   grid beats a lossy round-trip at a higher price). Pinned by
+   `TestGreedyReshopAfterOverflow` + the `cons_heavy_flat` /
+   `self_suff_flat_low_soc` simulator scenarios.
+
+10. **MILP discrete-extraction is cost-ranked (July 2026).** The MILP solves a
+   continuous LP, then collapses it to discrete full-power slots capped by
+   battery physics.  That collapse now executes the most cost-effective
+   subset:
+   - **Sells dearest-first.** When the cap allows fewer sell slots than the LP
+     spread energy across (the LP cycles PV — sell early to clear room, then
+     sell again at the peak — so the early cheap slot and the peak tie on
+     energy), the extraction keeps the **highest-price** slots.  Fixes MILP
+     selling a cheap 0.18 slot instead of the 0.40 evening peak
+     (`to_grid_sell_surplus`).
+   - **Charges cheapest-first, per day.** Charge selection is capped **per day**
+     at the energy the LP allocated to that day (preserving the midnight-reserve
+     "today-first" split — a single global cheapest-first cap would let cheap
+     tomorrow slots starve today), and within each day picks the cheapest slots.
+     A **half-slot rule** stops it pulling in a pricier slot to cover a
+     sub-half-slot remainder — which is how an expensive reserve-top-off slot
+     used to sneak in.  Fixes MILP charging a 0.30 evening peak to hit a high
+     self_consumption reserve (`self_suff_flat_low_soc`); it now charges the
+     0.15 slots and accepts the cheaper round-trip, like greedy.
+   Pinned by `test_milp_sells_evening_peak_not_cheap_early_slot`,
+   `test_milp_no_peak_charge_to_hit_reserve`, and the two simulator scenarios
+   (whose expectations now assert "no peak charge" / "sells the peak" for BOTH
+   engines).
 
 ---