[Bug]: Premature segment release and next-target churn can block multi-shard convergence

[weiliu1031]

Environment

• Milvus version: master
• Deployment mode: cluster
• Component: QueryCoord
• Collection layout: multiple shards

Current Behavior

QueryCoord can observe different readable target versions across shards
while the current and next targets continue to advance.

For example:

• current target: V1
• shard ch1 delegator: V2
• shard ch2 delegator: V3
• next target: V4

Three behaviors interact in this state.

1. Segments may be released before all delegators leave them

The segment release check only considers a subset of delegator versions
as potentially using an old sealed segment.

A delegator whose target version is ahead of current may still expose a
readable snapshot that references the segment, but the segment can be
treated as releasable.

This can release data that is still referenced by one of the shards.

2. Periodic next-target replacement can prevent convergence

TargetObserver force-refreshes an existing next target after
NextTargetSurviveTime, even when segment loading is still progressing.

If next changes from V4 to V5 before all delegators reach V4, and this
continues periodically, no target generation can fully synchronize and
be promoted to current.

After applying a conservative release barrier, this target churn would
also prevent old segments from ever becoming releasable.

3. ErrSegmentNotFound bypasses TargetObserver

When a segment load task returns ErrSegmentNotFound, the task scheduler
directly updates TargetManager.

This bypasses TargetObserver's target progress state, stale-target state,
and collection-level target update sequencing.

Expected Behavior

1. A sealed segment must remain in use while any delegator target version
   differs from the current target version.

2. A healthy next target must remain stable while the number of loaded
   target segments is increasing.

3. TargetObserver should force-refresh next only when:

   • no loading progress is observed for NextTargetSurviveTime; or
   • a load task explicitly reports that a target segment no longer
     exists.

4. A next target known to be stale must not be promoted to current.

5. Runtime next-target refreshes should be coordinated by TargetObserver
   instead of directly mutating TargetManager from the task scheduler.

Steps To Reproduce

1. Load a collection with at least two shards.
2. Let the shard delegators report different target versions.
3. Keep current behind both delegator versions.
4. Create a newer next target that has not fully converged.
5. Allow the periodic next-target timeout to replace next repeatedly.
6. Observe that:
   • an old segment may be released while a delegator still references
     it; or
   • conservative release remains blocked because current never catches
     up.

A load task returning ErrSegmentNotFound also demonstrates the direct
scheduler-to-TargetManager update path.

Proposed Fix

Fix 1: Make segment release conservative

Treat every delegator whose target version differs from current as
potentially using the segment.

Release the segment only after all relevant delegators converge to the
current target version.

Fix 2: Refresh next based on loading progress

Track the next target version and the number of unique next-target
segments present in SegmentDist.

When the version changes or the loaded segment count increases, record a
new progress baseline and restart the timeout.

Force-refresh next only when the count does not increase for
NextTargetSurviveTime.

Do not perform a timeout-based refresh after target synchronization has
started.

Fix 3: Mark an unloadable next target as stale

When the scheduler receives ErrSegmentNotFound, mark the collection's
next target stale instead of updating TargetManager directly.

On the next TargetObserver cycle:

• block stale next from being promoted to current;
• force-refresh next under the observer's collection lock;
• keep the stale marker when refresh fails;
• clear it after refresh succeeds.

How The Fixes Work Together

The conservative release rule provides the safety guarantee but can
temporarily retain more data on QueryNodes.

Progress-aware refresh gives a healthy target enough time to converge,
allowing current to advance and eventually clearing the release barrier.

The explicit stale-target signal provides the complementary recovery
path: a next target that cannot be loaded is refreshed without waiting
for the progress timeout.

Together, the changes provide:

• safe segment retention;
• stable target convergence;
• explicit recovery from invalid targets.

Risks

The conservative release barrier can temporarily increase QueryNode
memory usage.

If retained old segments leave insufficient capacity to load the next
target, current-target promotion and release can still block each other.
Additional QueryNode capacity may be required to allow convergence.