[kernel-spark] Support CDC + schema tracking log in v2

[PorridgeSwim]

🥞 Stacked PR

Use this link to review incremental changes.

• stack/SparkMetadataAdapter [Files changed] [MERGED]
  • stack/RefactorMetadataTrackingLog [Files changed] [MERGED]
    • stack/RefactorDeltaSourceMetadataEvolutionSupport [Files changed] [MERGED]
      • stack/MetadataEvolutionHandler2 [Files changed] [MERGED]
        • stack/NonAdditiveSchemaEvolution2 [Files changed] [MERGED]
          • stack/NonAdditiveSchemaEvolution3 [Files changed]
            • stack/consecutiveSchemaChangesMerger [Files changed]
              • stack/SchemaTrackingWithCDC [Files changed]
              • stack/V1V2MixTest [Files changed]

---

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

Follow-up to the non-additive schema evolution stack: extend V2 streaming schema tracking to CDC reads so a CDC stream stops at metadata- or protocol-change commits with a barrier sentinel instead of either silently reading across the change or failing the read-compat check.

• SparkMicroBatchStream.collectAndBuildCDCIndexedFiles: capture Protocol alongside Metadata while scanning a commit's actions, then call MetadataEvolutionHandler.getMetadataOrProtocolChangeIndexedFileIterator once the scan is done; when the commit diverges from source-init, return a singleton barrier (METADATA_CHANGE_INDEX) in place of BASE + files + END. Skip the on-commit verifyMetadataAction read-compat check when schema tracking is active — the barrier covers divergence. V1 splits this between DeltaSourceCDCSupport.filterAndIndexDeltaLogs (barrier injection) and IndexedChangeFileSeq.filterFiles (short-circuit); V2 collapses both into this single method.
• SparkMicroBatchStream.applyPerCommitCDCAdmission: pass barrier sentinels through admission unchanged (they can only appear as element 0 of the per-commit list).
• SparkMicroBatchStream.getFileChangesForCDC: apply metadataEvolutionHandler.stopIndexedFileIteratorAtSchemaChangeBarrier after end-boundary filtering so post-barrier commits in the same batch are truncated. V1's wrap lives in the shared DeltaSource.getFileChangesWithRateLimit; V2 places it inside the CDC-specific method because both planInputPartitions and the outer getFileChangesWithRateLimit reach the CDC iterator through here.
• MetadataEvolutionHandler.getMergedConsecutiveMetadataChanges: include AddCDCFile in the action set the merger walks and treat any non-null CDC column as a file action that stops the merger walk. Drops the mergeActionSet parameter (always CDC_ACTION_SET now) so non-CDC and CDC analysis share the same stop semantics. Resolves the TODO([Feature Request] Implement kernel-based dsv2 delta streaming source (M2: support advanced read options) #5319) placeholder left by PR 7/7.
• SparkMicroBatchStream.CDC_ACTION_SET: promote to public so MetadataEvolutionHandler can reuse it.

How was this patch tested?

SparkMicroBatchStreamCDCTest adds barrier-emission cases:

• testProcessCommit_emitsBarrierAtSchemaChange: a metadata-only commit on a CDC stream with seeded tracking emits [barrier, END] from processCommitToIndexedFilesForCDC.
• testGetFileChangesForCDC_emitsBarrierAtSchemaChange: end-to-end check that the barrier fires and the iterator truncates across commits — exercises metadata/protocol capture, barrier emission, admission passthrough, and cross-commit truncation in one path.

MetadataEvolutionHandlerTest extends the merger walk tests to cover CDC file actions stopping the walk. The unified DeltaV2SourceSchemaEvolutionCDCSuiteBase moves all evolution scenarios out of shouldFailTests into shouldPassTests so the CDC variants of the streaming schema-evolution suite now run alongside non-CDC.

Does this PR introduce any user-facing changes?

No.