[PROTOCOL] Add delta.parquet.compression.codec property to protocol #6324
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| # Parquet Compression Codec | ||
| **Associated Github issue for discussions: https://github.com/delta-io/delta/issues/6323 | ||
|
emkornfield marked this conversation as resolved.
Outdated
emkornfield marked this conversation as resolved.
Outdated
|
||
|
|
||
| Delta Lake tables store data in parquet files, and parquet supports multiple compression codecs. Currently, the Delta protocol does not formally document how compression is specified or which codecs are supported. This RFC introduces the `delta.parquet.compression.codec` table property to persistently configure the compression codec used for new parquet files. | ||
|
|
||
| -------- | ||
|
|
||
| > ***New top-level Section just before [Appendix](#appendix)*** | ||
|
|
||
| # Table Properties | ||
|
|
||
| Delta Lake tables support a set of properties stored in the `configuration` field of the `metaData` action that control various aspects of table behavior. | ||
|
|
||
| ## Overview | ||
|
|
||
| Property | Description | ||
| -|- | ||
| [`delta.parquet.compression.codec`](#deltaparquetcompressioncodec) | Compression codec for new Parquet data and checkpoint files | ||
|
|
||
| ## Property Details | ||
|
|
||
| ### delta.parquet.compression.codec | ||
|
Contributor
There was a problem hiding this comment. nit: this is a title. each property will be a title? without a table .. i am not sure how the list of properties will look like not a blocker for merging the RFC. we can refactor it when merging into the protocol as well. but i suggest following standards eventually. |
||
|
|
||
| Specifies the compression codec writers SHOULD use when writing new Parquet data and checkpoint files. Changing this property does not affect existing files; a table may contain files written with different codecs, which is a normal and expected state. | ||
|
|
||
| Supported values (matched case-insensitively): | ||
|
Collaborator
There was a problem hiding this comment. Should we clarify that this is a best-effort list? We don't definitively state the exhaustive list of supported values? In other words: is it VALID for a Delta table to have a DIFFERENT value?
Collaborator
Author
There was a problem hiding this comment. I think this is covered below on writer requirements?
Collaborator
There was a problem hiding this comment. Somewhat?
Is there a simple sentence or clause we can add that clears this up? removes the ambiguity?
emkornfield marked this conversation as resolved.
Outdated
|
||
|
|
||
| Value | Description | ||
| -|- | ||
| `uncompressed` or `none` | No compression | ||
| `snappy` | Snappy compression (recommended default) | ||
| `gzip` | GZIP compression | ||
| `lz4` | (Deprecated) LZ4 compression (Hadoop framing). For backwards compatibility only. | ||
|
emkornfield marked this conversation as resolved.
Outdated
|
||
| `lz4_raw` | [LZ4 compression](https://parquet.apache.org/docs/file-format/data-pages/compression/#lz4_raw) based on the LZ4 block compression format. | ||
|
emkornfield marked this conversation as resolved.
Outdated
emkornfield marked this conversation as resolved.
Outdated
|
||
| `zstd` | Zstandard compression | ||
|
|
||
| When the property is absent, writers SHOULD default to `zstd`. If a writer does not support the specified codec, it SHOULD abort with an appropriate error or fall back to a default codec. | ||
|
emkornfield marked this conversation as resolved.
Outdated
|
||
|
|
||
| Readers SHOULD be able to read parquet files compressed with any of the supported codecs, regardless of the current table property value. In some cases parquet files might have been written codecs that [parquet supports](https://parquet.apache.org/docs/file-format/data-pages/compression/) that are not in the list above, readers MAY support reading these files. | ||
|
Collaborator
There was a problem hiding this comment. been written *with codecs that ...
emkornfield marked this conversation as resolved.
Outdated
emkornfield marked this conversation as resolved.
Outdated
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hey @emkornfield ! I think for something like this you can just make a PR directly against PROTOCOL.md.
WDYT @tdas ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think so too. if its not a breaking change, just fully backward compatible improvements, then we can just add it to the protocol directly.