[PROTOCOL] Add delta.parquet.format.version property - #6539
Merged
scottsand-db merged 1 commit intoApr 10, 2026
Conversation
tdas
reviewed
Apr 10, 2026
tdas
approved these changes
Apr 10, 2026
sanujbasu
force-pushed
the
protocol-parquet-format-version
branch
from
April 10, 2026 15:35
8fd0797 to
a07722f
Compare
1/ Add the description of a property to the protocol. 2/ Add Table Properties entry to table of contents
sanujbasu
force-pushed
the
protocol-parquet-format-version
branch
from
April 10, 2026 15:51
0350416 to
e4c1fa4
Compare
scottsand-db
approved these changes
Apr 10, 2026
sanujbasu
added a commit
to sanujbasu/delta-kernel-rs
that referenced
this pull request
Apr 10, 2026
Add support for the delta.parquet.format.version table property so connectors can read it from TableProperties to configure their Parquet writers. Changes: - Add PARQUET_FORMAT_VERSION constant and parquet_format_version field to TableProperties with parsing in deserialize.rs - Add to ALLOWED_DELTA_PROPERTIES for CREATE TABLE support - Add tests for property parsing and CREATE TABLE validation Protocol spec: delta-io/delta#6539
github-merge-queue Bot
pushed a commit
to delta-io/delta-kernel-rs
that referenced
this pull request
Apr 13, 2026
## What changes are proposed in this pull request? Add support for the delta.parquet.format.version table property so connectors can read it from TableProperties to configure their Parquet writers. 1. Add PARQUET_FORMAT_VERSION constant and parquet_format_version: Option<String> field to TableProperties with parsing in deserialize.rs 2. Add to ALLOWED_DELTA_PROPERTIES for CREATE TABLE support 3. Add tests for property parsing and CREATE TABLE validation Protocol spec: delta-io/delta#6539 ## How was this change tested? 1. test_property_key_constants -- verifies the constant string value 2. test_parse_table_properties -- verifies parsing into TableProperties.parquet_format_version 3. test_parquet_format_version_accepted -- verifies the property passes CREATE TABLE validation and is preserved without triggering any feature flags 4. cargo clippy -p delta_kernel --all-features -- -D warnings passes clean Co-authored-by: Sanuj Basu <sanujbasu@users.noreply.github.com>
huashi-st
pushed a commit
to huashi-st/delta
that referenced
this pull request
Apr 24, 2026
## Summary 1/ Adds delta.parquet.format.version to the Table Properties section of the protocol spec 2/ Documents valid values (`1.0.0` for DataPageV1, `2.x.x` for DataPageV2), writer validation behavior, reader expectations, and cross-engine interop guidance ## Details 1/ Writers: use this property to select the Parquet data page format. Valid values are `1.0.0` (DataPageV1) and any `2.x.x` (DataPageV2). Recommended values are `1.0.0` and `2.12.0`. Writers SHOULD default to `1.0.0` when absent. 2/ Readers: do not need to consult this property. Parquet pages are self-describing via the `PageType` field in each page header. Readers SHOULD support both DataPageV1 and DataPageV2. 3/ Interop: tables accessed by engines beyond the Delta Lake connectors SHOULD use `1.0.0`, as DataPageV2 support varies across the broader Parquet ecosystem. ## Test plan - [x] Protocol-only change — no code changes, no tests needed - [x] Verify markdown table renders correctly Co-authored-by: Sanuj Basu <sanuj.basu@your.hostname.com>
Contributor
|
@sanujbasu, @scottsand-db, @tdas, @emkornfield I noticed that in the For the checkpoint, are we intentionally ignoring the |
Collaborator
This |
Merged
5 tasks
Contributor
|
Here is the PR to update protocol: #6797 After that I'll check the required code changes |
tdas
pushed a commit
that referenced
this pull request
May 22, 2026
…e checkpoint files (#6797) #### Which Delta project/connector is this regarding? - [ ] Spark - [ ] Standalone - [ ] Flink - [ ] Kernel - [X] Other (Protocol) ## Description Fix this comment: #6539 (comment) ## How was this patch tested? N/A ## Does this PR introduce _any_ user-facing changes? Yes, protocol clarity Signed-off-by: Felipe Fujiy Pessoto <fepessot@microsoft.com> Co-authored-by: Scott Haines <newfrontcreative@gmail.com> Co-authored-by: Timothy Wang <timothy.art@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
1/ Adds delta.parquet.format.version to the Table Properties section of the protocol spec
2/ Documents valid values (
1.0.0for DataPageV1,2.x.xfor DataPageV2), writer validation behavior, reader expectations, and cross-engine interop guidanceDetails
1/ Writers: use this property to select the Parquet data page format. Valid values are
1.0.0(DataPageV1) and any2.x.x(DataPageV2). Recommended values are1.0.0and2.12.0. Writers SHOULD default to1.0.0when absent.2/ Readers: do not need to consult this property. Parquet pages are self-describing via the
PageTypefield in each page header. Readers SHOULD support both DataPageV1 and DataPageV2.3/ Interop: tables accessed by engines beyond the Delta Lake connectors SHOULD use
1.0.0, as DataPageV2 support varies across the broader Parquet ecosystem.Test plan