Porter can strand parcels until daemon restart

[jtobin]

(NB, this is the leading hypothesis for the remaining flake noted in #1976, i.e. "missing anchor block hash, transfer not confirmed." The actual bug itself appears to be in chain porter's error handling; the "primary" and "tertiary" potential fixes IMO look most promising.)

(EDIT: this has been ruled out as the cause of #1976.)

Failure

CI run 25216477057, branch mssmt-unit-db-backends @
657fd71. Both htlc_force_close and htlc_force_close_mpp
fail with:

missing anchor block hash, transfer not confirmed

at locateAssetTransfers for Alice's final second-level HTLC
success sweep (helpers.go:3155; CI shows :3173 due to line
drift on the CI branch). The transfer row exists; the block
hash does not. The test polls for 2 minutes and times out.

This is NOT the timeout-sweep mempool/RBF flake fixed by
3d8d534. That failure occurs later in the test; this one
fires in the HTLC success path before timeout sweeps begin.

Observed State

• ListTransfers returns exactly one transfer for the mined
  sweep txid.
• AnchorTxBlockHash is nil for the full poll window.
• The predecessor sweep (first-level HTLC success tx) was
  already confirmed: locateAssetTransfers at
  helpers.go:3122 succeeded, meaning storeProofs and
  LogAnchorTxConfirm both completed for it.

Transfer Lifecycle

StorePreBroadcast     LogPendingParcel — row written
Broadcast             PublishTransaction
WaitTxConf            deliverOutboundPkgResp — caller unblocked
                      waitForTransferTxConf  — blocks for conf
StorePostAnchorTxConf storeProofs
                      storePackageAnchorTxConf
                        → LogAnchorTxConfirm — block hash written

The row exists but the block hash was never written. The state
machine stopped advancing somewhere between
deliverOutboundPkgResp and LogAnchorTxConfirm.

Root Cause

Structural bug: silent post-delivery error

The porter delivers the response to the caller in WaitTxConf
(chain_porter.go:2096) before doing the confirmation wait
and post-confirmation work. RequestShipment returns on
respChan; NotifyBroadcast returns to lnd's sweeper. From
that point the porter goroutine is unsupervised.

Any subsequent error goes to errChan, which nobody reads:

// chain_porter.go:380-384
updatedPkg, err := p.stateStep(*pkg)
if err != nil {
    kit.errChan <- err  // buffered(1), no reader
    log.Errorf(...)
    return              // goroutine exits
}

The error is logged at ERROR level in tapd's daemon logs (not
captured in CI test output). The goroutine exits. The parcel
stays pending. LogAnchorTxConfirm never runs.
resumePendingParcels would rescue it on restart, but tests
don't restart tapd.

This is the decisive structural bug. Without it, any transient
failure would propagate to the caller and the sweeper would
retry.

Likely trigger: confirmation/finalization failure

The specific error that fires within the silent-failure window
is not directly observable from the CI test logs (only tapd
ERROR logs would show it). The candidates, ranked:

1. Confirmation not delivered. The porter registers for
   confirmation via RegisterConfirmationsNtfn with a height
   hint captured before broadcast (aux_sweeper.go:2579).
   The registration includes a historical rescan, which should
   catch already-confirmed txs. But there is a known class of
   race in chain notifiers: if the confirming block is
   processed between the scan completing and the watch being
   installed, the notification is lost. This race is narrow
   but widens under CI load (many parallel tests, rapid block
   mining). Commit eea18cf ("add sweep shipment height
   hint") was a targeted mitigation for this area, confirming
   it was already known to be timing-sensitive.

2. storeProofs or storePackageAnchorTxConf failure.
   If confirmation IS delivered but a subsequent step fails
   (proof fetch, verification, DB update), the same
   silent-error path strands the parcel.

3. Broadcast state failure / stale parcel. The porter
   redundantly calls PublishTransaction for sweep txs that
   lnd also broadcasts. A transient gRPC failure could leave
   an orphaned stored-but-unprocessed parcel. RBF handling
   is unimplemented (aux_sweeper.go:2397 TODO). Less likely
   for this specific run (single broadcast, valid tx, no
   fee-bumping).

Demoted hypothesis: proof archive race

An earlier hypothesis proposed that storeProofs for the
second-level sweep fails because fetchInputProof can't find
the first-level sweep's proof (not yet archived). This was
ruled out: locateAssetTransfers already succeeded for the
first-level sweep at helpers.go:3122, which requires both
storeProofs and LogAnchorTxConfirm to have completed. The
predecessor proof IS archived before the second-level sweep
begins. This remains a valid class of failure but is not the
trigger here.

Why Only the Second-Level Sweep

The second-level sweep is uniquely vulnerable:

• Longest dependency chain (commitment → first-level →
  second-level), so the most intermediate porter state to
  traverse before confirmation registration.
• Occurs latest in the test, after the most blocks have been
  mined, giving the most opportunity for scheduling jitter.
• assertSweepExists only confirms the sweep input is
  registered with the sweeper, not that the sweep tx has
  been broadcast or that the porter has registered for
  confirmation. The test may mine the confirming block before
  the porter reaches RegisterConfirmationsNtfn.

Fix

Primary: make post-delivery failures recoverable

The silent-error pattern is the structural enabler. Fix it:

• Retry in-process. When waitForTransferTxConf,
  storeProofs, or storePackageAnchorTxConf fails after
  deliverOutboundPkgResp, retry with backoff rather than
  exiting.

• Background recovery. Periodically check for parcels
  whose anchor tx is confirmed on-chain but whose block hash
  is nil. Re-enter StorePostAnchorTxConf for them. This
  covers both "confirmation never delivered" (the tx IS
  confirmed, the porter just didn't hear) and "post-
  confirmation processing failed." Also provides production
  resilience without requiring a daemon restart.

Secondary: skip broadcast for sweep parcels

Set skipAnchorTxBroadcast=true in shipChannelTxn
(aux_closer.go:682). Infrastructure already exists. Since
lnd's sweeper handles broadcasting, the porter's
PublishTransaction is redundant. Skipping it eliminates the
Broadcast failure mode and shortens the window between parcel
storage and confirmation registration. Requires moving
importLocalAddresses to before the skip check in
StorePreBroadcast.

Tertiary: tighten confirmation registration

Register for confirmation earlier in the pipeline (e.g., in
StorePreBroadcast rather than after deliverOutboundPkgResp
in WaitTxConf), so the watch is active before the tx is
published and the race window is minimised.

Key Files

File	Line	Role
tapfreighter/chain_porter.go	380	advanceState — unread errChan
tapfreighter/chain_porter.go	416	waitForTransferTxConf
tapfreighter/chain_porter.go	886	LogAnchorTxConfirm
tapfreighter/chain_porter.go	2013	LogPendingParcel
tapfreighter/chain_porter.go	2024	SkipAnchorTxBroadcast check
tapfreighter/chain_porter.go	2042	Broadcast — redundant publish
tapfreighter/chain_porter.go	2096	deliverOutboundPkgResp
tapfreighter/chain_porter.go	2104	StorePostAnchorTxConf
tapchannel/aux_sweeper.go	2393	registerAndBroadcastSweep
tapchannel/aux_sweeper.go	2575	height hint (best-effort)
tapchannel/aux_closer.go	682	skipAnchorTxBroadcast=false
helpers.go	3122	predecessor sweep confirmed OK
helpers.go	3155	failing call

[kaldun-tech]

Observed this in a CI failure https://github.com/lightninglabs/taproot-assets/actions/runs/26522178163/job/78267541273 for PR #2130

[jtobin]

Closing as obviated by #2159 and #2160.