You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "emkornfield (via GitHub)" <gi...@apache.org> on 2025/12/02 18:18:09 UTC
[PR] GH-534: Clarify versioning and V2 [parquet-format]
emkornfield opened a new pull request, #535:
URL: https://github.com/apache/parquet-format/pull/535
### Rationale for this change
Questions still arise on the state of V2, this tries to clarify my current understanding.
1. All encodings can be used any place
2. Version in the footer is meaningless and readers should determine readability by features.
### What changes are included in this PR?
Spec clarification. Will start a discussion on the mailing list for word-smithing and consensus
### Do these changes have PoC implementations?
No
Closes #534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #535:
URL: https://github.com/apache/parquet-format/pull/535#issuecomment-3672986069
I also added a links to the related mailing list threads to the description on this PR:
- related mailing list thread https://lists.apache.org/thread/0bdyyb7qobrxx94x8v7t5z7g2ksnpyr2
- vote thread on mailing list: https://lists.apache.org/thread/1nf596dq7fpc0zp35bt17zy7srvo5wt8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2597405446
##########
src/main/thrift/parquet.thrift:
##########
@@ -715,6 +715,13 @@ struct DictionaryPageHeader {
* New page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ *
+ * As of December 2025, most known Parquet readers can read pages using this header.
Review Comment:
rehrased along these lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2582449315
##########
src/main/thrift/parquet.thrift:
##########
@@ -1255,7 +1259,12 @@ union EncryptionAlgorithm {
* Description for file metadata
*/
struct FileMetaData {
- /** Version of this file **/
+ /** Version of this file
+ *
+ * Deprecated. Readers should determine if they support reading based on
+ * specific metadata (e.g. encoding enum) rather then relying on this field
+ * to make this determination.
+ */
Review Comment:
I disagree with this. I don't think we should abandon versioning, but rather be more explicit about breaking changes and what is included with version update. Regardless, this needs more discussion with the community and a clear path forward for how we support breaking changes.
##########
src/main/thrift/parquet.thrift:
##########
@@ -715,6 +715,10 @@ struct DictionaryPageHeader {
* New page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * N.B. this page header is not necessarily strictly better then DataPageHeader.
+ * Page indexes already require that rows are aligned on page boundaries, and compressing
+ * repetition and definition levels can still be effective in some cases.
Review Comment:
Are you saying this is deprecated? Why do we need this comment? It's not clear what you're trying to achieve here. (Nit: prefer not to use abbreviations like N.B.)
##########
Encodings.md:
##########
@@ -22,6 +22,11 @@ Parquet encoding definitions
This file contains the specification of all supported encodings.
+Some Parquet implementations distinguish encodings as "v1" and "v2". From
+a specification perspective this distinction is considered meaningless. Writers may use any
+encoding with both data page v1 and data page v2. Readers should lazily evaluate if they can
+read a file (e.g. only error when required to a read a page with an unknown encoding).
+
Review Comment:
I feel like we're redefining what `version` means to be scoped only to encodings and then saying that it's not necessary. It seems like we want to either separate encodings from versioning (e.g. any encoding that is understood by a client should be considered supported regardless of when it was introduced) or be more explicit about associating new encodings with a version (along with other possible breaking structural/representational changes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2633261774
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,14 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
Review Comment:
> the naming of "V2" somewhat implies that it came after DataPageHeader
Right, IIUC the concern from Antoine is to maybe note that this wasn't part of the original spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #535:
URL: https://github.com/apache/parquet-format/pull/535#issuecomment-3625433610
> I do think we should make sure that @danielcweeks 's concerns about version number are addressed before merging this
I discussed the changes with @danielcweeks offline and I think he is OK with the current state. @danielcweeks please let me know if I misunderstood.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2631762258
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,14 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
Review Comment:
I would perhaps keep "New" or "More recent".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #535:
URL: https://github.com/apache/parquet-format/pull/535#issuecomment-3675740016
Vote passed. Going to merge. @pitrou @alamb we can maybe take another pass at the V2 data page header in another PR if there are serious concerns here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2616751178
##########
Encodings.md:
##########
@@ -59,8 +62,8 @@ Dictionary page format: the entries in the dictionary using the [plain](#PLAIN)
Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32),
followed by the values encoded using RLE/Bit packed described above (with the given bit width).
-Using the PLAIN_DICTIONARY enum value is deprecated in the Parquet 2.0 specification. Prefer using RLE_DICTIONARY
-in a data page and PLAIN in a dictionary page for Parquet 2.0+ files.
+Using the `PLAIN_DICTIONARY` enum value is deprecated, prefer using `RLE_DICTIONARY`.
+in a data page and PLAIN in a dictionary page for new Parquet files.
Review Comment:
removed the word prefer. I think this probably strikes the right balance.
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,16 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ * https://parquet.apache.org/docs/file-format/implementationstatus/ tracks the implementation of readers and
+ * writers that support this page format.
Review Comment:
Removed. @alamb suggested adding it but I agree we generally don't reference it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2589924160
##########
src/main/thrift/parquet.thrift:
##########
@@ -1255,7 +1259,12 @@ union EncryptionAlgorithm {
* Description for file metadata
*/
struct FileMetaData {
- /** Version of this file **/
+ /** Version of this file
+ *
+ * Deprecated. Readers should determine if they support reading based on
+ * specific metadata (e.g. encoding enum) rather then relying on this field
+ * to make this determination.
+ */
Review Comment:
I rephrased this to explain rationale. I agree it needs discussion in the community but I think based on previous conversations this is probably the consensus path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2594172712
##########
src/main/thrift/parquet.thrift:
##########
@@ -1255,7 +1262,12 @@ union EncryptionAlgorithm {
* Description for file metadata
*/
struct FileMetaData {
- /** Version of this file **/
+ /** Version of this file
+ *
+ * Writers should always populate "1" for version. Readers should
+ * accept "1" and "2" interchangeably. All other versions are Currently
Review Comment:
```suggestion
* accept "1" and "2" interchangeably. All other versions are currently
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2609918350
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,16 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ * https://parquet.apache.org/docs/file-format/implementationstatus/ tracks the implementation of readers and
+ * writers that support this page format.
Review Comment:
It does not seem useful to provide this link here as it would apply to most other Parquet features as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2631764452
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,14 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
Review Comment:
```suggestion
* Implementation note - this header is not necessarily a strict improvement over
* `DataPageHeader` (in particular the original header might provide better compression
* in some scenarios). Page indexes require pages to start and end at row boundaries,
* regardless of which page header is used.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2604129364
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,16 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ * https://parquet.apache.org/docs/file-format/implementationstatus/ tracks the implementation of readers and
Review Comment:
```suggestion
* https://parquet.apache.org/docs/file-format/implementationstatus/ tracks the implementation of readers and
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2598258576
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,16 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ * https://parquet.apache.org/docs/file-format/implementationstatus/ tracks implementation readers and
+ * writers that support this version.
Review Comment:
```suggestion
* https://parquet.apache.org/docs/file-format/implementationstatus/ tracks the implementation of readers and
* writers that support this page format.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2610897605
##########
Encodings.md:
##########
@@ -59,8 +62,8 @@ Dictionary page format: the entries in the dictionary using the [plain](#PLAIN)
Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32),
followed by the values encoded using RLE/Bit packed described above (with the given bit width).
-Using the PLAIN_DICTIONARY enum value is deprecated in the Parquet 2.0 specification. Prefer using RLE_DICTIONARY
-in a data page and PLAIN in a dictionary page for Parquet 2.0+ files.
+Using the `PLAIN_DICTIONARY` enum value is deprecated, prefer using `RLE_DICTIONARY`.
+in a data page and PLAIN in a dictionary page for new Parquet files.
Review Comment:
```suggestion
Using the `PLAIN_DICTIONARY` enum value is deprecated, prefer using `RLE_DICTIONARY`
in a data page and `PLAIN` in a dictionary page for new Parquet files.
```
BTW, should we change `prefer using` to `should always use`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2632879285
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,14 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
Review Comment:
I think Alternate probably makes the most sense here. It is not exactly "new" any more.
How about "Alternative page format, introduced to the specification after DataPageHeader, ...."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2589921092
##########
Encodings.md:
##########
@@ -22,6 +22,11 @@ Parquet encoding definitions
This file contains the specification of all supported encodings.
+Some Parquet implementations distinguish encodings as "v1" and "v2". From
+a specification perspective this distinction is considered meaningless. Writers may use any
+encoding with both data page v1 and data page v2. Readers should lazily evaluate if they can
+read a file (e.g. only error when required to a read a page with an unknown encoding).
+
Review Comment:
Rephrased, and explained rationale, I agree this needs discussion with the community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "danielcweeks (via GitHub)" <gi...@apache.org>.
danielcweeks commented on PR #535:
URL: https://github.com/apache/parquet-format/pull/535#issuecomment-3613658184
After thinking this through a little more, I think we should more clearly define what each "versioned identifier" means and clearly articulate under what conditions it would change. For example:
## Magic Number `PAR1`:
- What it means: Indicates that footer is still thrift compatible with Parquet V1 format and is expected to be parseable by any existing Parquet V1 client.
- When it changes: the footer is changed in an incompatible way with the thrift definition or the footer is substantively changed in a way that older clients should not even attempt to read the file.
- What it solves: helps determine whether a file/footer is parquet/corrupt/other or substantively different between versions.
- Example: replacing the footer with FlatBuffers representation in an incompatible way.
## Footer Version Number
- What does it mean: largely redundant with PAR1
- When it changes: (same as magic number)
- What it solves: if the footer is stored somewhere outside of the file (e.g. a cache or supplied via some other mechanism)
- Example: the footer is stored in a high-performance cache and keyed by path for faster pruning. No magic number is available for the reader.
## What changes have been made _without version updates_:
1. [backward __incompatible__] New compression codecs: brotli, zstandard, etc.
2. [backward __incompatible__] New data types: variant, geo types, etc.
3. [backward compatible] Addition of Page indexes
4. [backward compatible] Updates to Column Stats
Given the incompatible changes like addition of codecs without a version change, it's confusing as to why the addition of encodings would require a version change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2597404365
##########
src/main/thrift/parquet.thrift:
##########
@@ -1255,7 +1259,12 @@ union EncryptionAlgorithm {
* Description for file metadata
*/
struct FileMetaData {
- /** Version of this file **/
+ /** Version of this file
+ *
+ * Deprecated. Readers should determine if they support reading based on
+ * specific metadata (e.g. encoding enum) rather then relying on this field
+ * to make this determination.
+ */
Review Comment:
rephrased along these lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #535:
URL: https://github.com/apache/parquet-format/pull/535#issuecomment-3625462913
@alamb thanks for the review, I believe I addressed your concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2597981291
##########
src/main/thrift/parquet.thrift:
##########
@@ -1255,7 +1262,14 @@ union EncryptionAlgorithm {
* Description for file metadata
*/
struct FileMetaData {
- /** Version of this file **/
+ /** Version of this file
+ *
+ * As of December 2025, there is no agreed upon consensus of what constitutes
+ * version 2 of the file. For maximum compatibility with readers, writers should
+ * always populate "1" for version. For maximum with writers compatibility,
Review Comment:
```suggestion
* always populate "1" for version. For maximum compatibility with writers,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2633204616
##########
src/main/thrift/parquet.thrift:
##########
@@ -712,9 +712,14 @@ struct DictionaryPageHeader {
}
/**
- * New page format allowing reading levels without decompressing the data
+ * Alternate page format allowing reading levels without decompressing the data
Review Comment:
the naming of "V2" somewhat implies that it came after `DataPageHeader`
What I think is confusing about the word "new" is that (to me) it implies an expectation that this header will eventually replace the existing one, which I don't think there is consensus on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2594291019
##########
src/main/thrift/parquet.thrift:
##########
@@ -1255,7 +1259,12 @@ union EncryptionAlgorithm {
* Description for file metadata
*/
struct FileMetaData {
- /** Version of this file **/
+ /** Version of this file
+ *
+ * Deprecated. Readers should determine if they support reading based on
+ * specific metadata (e.g. encoding enum) rather then relying on this field
+ * to make this determination.
+ */
Review Comment:
Rather than potentially (implicitly) changing the spec, I suggest we add some langague to help people understand how to interpret this field and the tradeoffs between using different versions.
For example, maybe we could say something like
> As of December 2025, there is no agreed upon set of features that constitute version 2, so for maximum compatibility, writers should populate "1" for version and accept "1" and "2" interchangeably.
##########
src/main/thrift/parquet.thrift:
##########
@@ -715,6 +715,13 @@ struct DictionaryPageHeader {
* New page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ *
+ * As of December 2025, most known Parquet readers can read pages using this header.
Review Comment:
Maybe we can also change the content of
> * New page format allowing reading levels without decompressing the data
To another word to make it clearer that there is no agreed upon expectation that all writers will eventually use this new header. Perhaps:
> * Alternate page format allowing reading levels without decompressing the data
##########
src/main/thrift/parquet.thrift:
##########
@@ -715,6 +715,13 @@ struct DictionaryPageHeader {
* New page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * `DataPageHeader` (in particular the original header might provide better compression
+ * in some scenarios). Page indexes require pages start and end at row boundaries regardless of which
+ * page header is used.
+ *
+ * As of December 2025, most known Parquet readers can read pages using this header.
Review Comment:
Perhaps we could refer readers to https://parquet.apache.org/docs/file-format/implementationstatus/ for the most up to date information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2589943105
##########
src/main/thrift/parquet.thrift:
##########
@@ -715,6 +715,10 @@ struct DictionaryPageHeader {
* New page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * N.B. this page header is not necessarily strictly better then DataPageHeader.
+ * Page indexes already require that rows are aligned on page boundaries, and compressing
+ * repetition and definition levels can still be effective in some cases.
Review Comment:
Rephrased, no I'm wasn't just saying it is not necessarily a strict improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2594172149
##########
src/main/thrift/parquet.thrift:
##########
@@ -715,6 +715,13 @@ struct DictionaryPageHeader {
* New page format allowing reading levels without decompressing the data
* Repetition and definition levels are uncompressed
* The remaining section containing the data is compressed if is_compressed is true
+ *
+ * Implementation note - this header is not necessarily a strict improvement over
+ * DataPageHeader in all cases (in particular the original header might provide better compression
Review Comment:
```suggestion
* `DataPageHeader` (in particular the original header might provide better compression
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2589956219
##########
Encodings.md:
##########
@@ -22,6 +22,11 @@ Parquet encoding definitions
This file contains the specification of all supported encodings.
+Some Parquet implementations distinguish encodings as "v1" and "v2". From
+a specification perspective this distinction is considered meaningless. Writers may use any
+encoding with both data page v1 and data page v2. Readers should lazily evaluate if they can
+read a file (e.g. only error when required to a read a page with an unknown encoding).
+
Review Comment:
I rephrased to remove version entirely as it might cause confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2589921092
##########
Encodings.md:
##########
@@ -22,6 +22,11 @@ Parquet encoding definitions
This file contains the specification of all supported encodings.
+Some Parquet implementations distinguish encodings as "v1" and "v2". From
+a specification perspective this distinction is considered meaningless. Writers may use any
+encoding with both data page v1 and data page v2. Readers should lazily evaluate if they can
+read a file (e.g. only error when required to a read a page with an unknown encoding).
+
Review Comment:
Rephrased, and explained rationale, I agree this needs discussion with the community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #535:
URL: https://github.com/apache/parquet-format/pull/535#discussion_r2604128328
##########
Encodings.md:
##########
@@ -59,8 +62,8 @@ Dictionary page format: the entries in the dictionary using the [plain](#PLAIN)
Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32),
followed by the values encoded using RLE/Bit packed described above (with the given bit width).
-Using the PLAIN_DICTIONARY enum value is deprecated in the Parquet 2.0 specification. Prefer using RLE_DICTIONARY
-in a data page and PLAIN in a dictionary page for Parquet 2.0+ files.
+Using the PLAIN_DICTIONARY enum value is deprecated. Prefer using RLE_DICTIONARY
Review Comment:
```suggestion
Using the `PLAIN_DICTIONARY` enum value is deprecated, prefer using `RLE_DICTIONARY`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-534: Clarify versioning and V2 [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield merged PR #535:
URL: https://github.com/apache/parquet-format/pull/535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org