[Spark] Fix streaming schema merger dropping metadata change when chain ends with a protocol-only commit

[PorridgeSwim]

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Description

DeltaSourceMetadataEvolutionSupport.getMergedConsecutiveMetadataChanges looks ahead through the run of consecutive non-file commits after the current persisted schema entry and writes a single merged PersistedMetadata covering the whole run. It used iteratorLast to pick only the final commit's actions, then took its Metadata/Protocol (or fell back to the previously persisted entry).

That breaks for chains shaped like (..., Metadata, Protocol-only): the final commit has no Metadata action, so metadataOpt.getOrElse(currentMetadata.dataSchemaJson) reverts to the pre-chain schema while deltaCommitVersion advances to the protocol-only tail. The persisted entry's dataSchema then disagrees with the snapshot at deltaCommitVersion, silently dropping the earlier metadata change.

This PR replaces the last-tuple logic with a fold across the chain, tracking the last version, the latest non-None Metadata action, and the latest non-None Protocol action separately. The merged entry now reflects the cumulative state at the end of the chain regardless of which commit each action lives in.

How was this patch tested?

Added a regression test consecutive schema evolutions with protocol-only tail in DeltaSourceSchemaEvolutionSuite. The chain is constructed so the start schema, the post-first-change schema (which becomes currentMetadata for the merger), and the post-chain schema are pairwise distinct, so a coincidence can't mask the failure:

• <a, b> → rename b→c → addColumn d → dropColumn c → protocol-only upgrade → file action.
• Pre-fix: merged entry has (vTail, <a, c>) — schema reverts to pre-chain.
• Post-fix: merged entry has (vTail, <a, d>) — schema reflects the final metadata change.

The existing consecutive schema evolutions test (all-Metadata chain) still passes since the fold's "latest metadata in chain" coincides with iteratorLast's result when every commit carries a Metadata action.

Does this PR introduce any user-facing changes?

No (bug fix to internal streaming schema-tracking behavior; no API changes).