feat: add index segment_seq metadata

[jackye1995]

Adds optional, name-scoped segment_seq metadata to index segments and assigns it in the commit/rebase layer. The change stores LogicalIndexMetadata.max_segment_seq in the manifest index section so sequence numbers are not reused after the highest physical segment for an index name is removed.

This backfills legacy segments on the next segment commit, uses the single writer feature flag FLAG_INDEX_SEGMENT_SEQ for both physical segment_seq and logical high-water metadata, and preserves existing Rust/Python/Java describe/build call patterns by assigning sequence numbers at commit time.

Discussion: #7044

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

❌ Patch coverage is 95.39317% with 58 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/transaction.rs	90.69%	16 Missing and 8 partials ⚠️
rust/lance/src/io/commit.rs	71.42%	12 Missing and 10 partials ⚠️
rust/lance-table/src/format/index.rs	95.06%	3 Missing and 1 partial ⚠️
rust/lance/src/dataset/optimize/remapping.rs	25.00%	3 Missing ⚠️
rust/lance/src/io/commit/conflict_resolver.rs	95.89%	1 Missing and 2 partials ⚠️
rust/lance-table/src/io/commit.rs	75.00%	0 Missing and 1 partial ⚠️
rust/lance/src/dataset.rs	94.73%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[LuQQiu]

Several high level comments

1. this PR does not have max_segment_seq, is it expected?
2. backfill when new segment arrive. Do we want to support backfill when no new segment arrive? how about re-commit with exact old segments == no op commit but also trigger backfill?

[wjones127]

The overall idea seems fine, but I have a couple of blocking questions

[wjones127]

issue(blocking): FLAG_INDEX_SEGMENT_SEQ_HIGH_WATER isn't documented in the protobuf.

Also, do we even need two separate flags? It seems like we could just have one flag for this and require both to be set. What do you think?

[jackye1995]

Done. I collapsed this back to a single writer flag: FLAG_INDEX_SEGMENT_SEQ now covers both physical IndexMetadata.segment_seq and logical LogicalIndexMetadata.max_segment_seq, and FLAG_UNKNOWN is back to 128.

The format docs and the vote/design discussion were updated with the single-flag compatibility model.

[wjones127]

praise: I'm in favor of adding this. It's been a long-time coming.

Eventually, we should move index details over to this, so there's just one copy per index.

Also, people have requested other fields like a top-level UUID and created_at date. I think we can add those in a future follow up.

[wjones127]

question(blocking): It's not clear to me from these test—how does a user opt-in to this segment sequence flag? We can't enable it by default, because that means that users can't downgrade their Lance version—any previous Lance library will (or should) error when trying to write to it, given it won't recognize the writer feature flags.

[jackye1995]

The flag is not enabled by a plain dataset write. It is set only once the manifest contains committed index segment sequence metadata, either physical segment_seq or logical max_segment_seq.

So the behavior is:

• Existing datasets without this metadata keep the flag clear and remain writable by older writers.
• A new writer enables the flag when it commits index segment metadata and writes segment_seq / logical_indexes.
• After that point, older writers should reject writes because they do not understand the writer feature flag and could otherwise drop the metadata.

I also added a test assertion that a freshly written dataset has FLAG_INDEX_SEGMENT_SEQ clear before index segments are committed, and set after segment commit.