You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "d33bs (via GitHub)" <gi...@apache.org> on 2026/08/22 18:55:11 UTC
[PR] GH-404: Clarify dictionary fallback encoding [parquet-format]
d33bs opened a new pull request, #609:
URL: https://github.com/apache/parquet-format/pull/609
### Rationale for this change
The dictionary encoding docs currently say that writers fall back to PLAIN encoding when a dictionary gets too large.
That is too narrow. Writers can stop using dictionary encoding and write later data pages with another valid encoding. The actual encoding is already stored in each data page header, so readers should use that field instead of assuming that fallback pages are PLAIN.
### What changes are included in this PR?
This PR updates `Encodings.md` to clarify that dictionary fallback encoding is chosen by the writer and recorded in the data page header.
It also clarifies that the documented dictionary data page layout applies only to dictionary-encoded data pages.
### Do these changes have PoC implementations?
I feel no PoC implementation is needed, but please let me know if you think otherwise. This is a documentation clarification for behavior that the format already supports.
Closes #404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-404: Clarify dictionary fallback encoding [parquet-format]
Posted by "d33bs (via GitHub)" <gi...@apache.org>.
d33bs commented on code in PR #609:
URL: https://github.com/apache/parquet-format/pull/609#discussion_r4076971749
##########
Encodings.md:
##########
@@ -77,13 +77,17 @@ endian integer, followed by the bytes.
The dictionary encoding builds a dictionary of values encountered in a given column. The
dictionary will be stored in a dictionary page per column chunk. The values are stored as integers
using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size
-or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is
-written first, before the data pages of the column chunk.
+or number of distinct values, the writer may stop using dictionary encoding and fall back to another
+valid encoding for subsequent data pages. The fallback encoding is chosen by the writer and recorded
+in the data page header's `encoding` field; readers must use that field to determine whether each
+data page is dictionary encoded or uses another encoding. The dictionary page is written first,
+before the data pages of the column chunk.
Review Comment:
Thanks @divjotarora ! I made an update which I hope reflects your apt comment's concerns. Let me know if you feel we should adjust further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-404: Clarify dictionary fallback encoding [parquet-format]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #609:
URL: https://github.com/apache/parquet-format/pull/609#discussion_r4085456940
##########
Encodings.md:
##########
@@ -76,14 +76,19 @@ endian integer, followed by the bytes.
### Dictionary Encoding (PLAIN_DICTIONARY = 2 and RLE_DICTIONARY = 8)
The dictionary encoding builds a dictionary of values encountered in a given column. The
dictionary will be stored in a dictionary page per column chunk. The values are stored as integers
-using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size
-or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is
-written first, before the data pages of the column chunk.
+using the [RLE/Bit-Packing Hybrid](#RLE) encoding. Within a column chunk, the dictionary page is
+written first, before the data pages of the column chunk. After the dictionary page, data pages can
+use dictionary encoding or another valid encoding for the column's data type; for example, a writer
+may stop using dictionary encoding if the dictionary grows too big, whether in size or number of
+distinct values. Dictionary-encoded and non-dictionary-encoded data pages may be interleaved within
Review Comment:
I mentioned it in my previous comment because the format already currently allows it (i.e. it's not explicitly banned). If readers don't support it, that's a bug IMO. I don't think it should block this PR specifically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-404: Clarify dictionary fallback encoding [parquet-format]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #609:
URL: https://github.com/apache/parquet-format/pull/609#issuecomment-5797656060
I don't think this will be controversial but we should probably send an e-mail to the dev@ list to let people know about the clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-404: Clarify dictionary fallback encoding [parquet-format]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #609:
URL: https://github.com/apache/parquet-format/pull/609#discussion_r4085385818
##########
Encodings.md:
##########
@@ -76,14 +76,19 @@ endian integer, followed by the bytes.
### Dictionary Encoding (PLAIN_DICTIONARY = 2 and RLE_DICTIONARY = 8)
The dictionary encoding builds a dictionary of values encountered in a given column. The
dictionary will be stored in a dictionary page per column chunk. The values are stored as integers
-using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size
-or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is
-written first, before the data pages of the column chunk.
+using the [RLE/Bit-Packing Hybrid](#RLE) encoding. Within a column chunk, the dictionary page is
+written first, before the data pages of the column chunk. After the dictionary page, data pages can
+use dictionary encoding or another valid encoding for the column's data type; for example, a writer
+may stop using dictionary encoding if the dictionary grows too big, whether in size or number of
+distinct values. Dictionary-encoded and non-dictionary-encoded data pages may be interleaved within
Review Comment:
Do we know if any writers actually interleave dictionary encoded pages with non dictionary pages? Do readers support this?
This seems a reasonable statement to me, but I am not sure if it expanding the range of data that is valid parquet 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-404: Clarify dictionary fallback encoding [parquet-format]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #609:
URL: https://github.com/apache/parquet-format/pull/609#discussion_r3890402466
##########
Encodings.md:
##########
@@ -77,13 +77,17 @@ endian integer, followed by the bytes.
The dictionary encoding builds a dictionary of values encountered in a given column. The
dictionary will be stored in a dictionary page per column chunk. The values are stored as integers
using the [RLE/Bit-Packing Hybrid](#RLE) encoding. If the dictionary grows too big, whether in size
-or number of distinct values, the encoding will fall back to the plain encoding. The dictionary page is
-written first, before the data pages of the column chunk.
+or number of distinct values, the writer may stop using dictionary encoding and fall back to another
+valid encoding for subsequent data pages. The fallback encoding is chosen by the writer and recorded
+in the data page header's `encoding` field; readers must use that field to determine whether each
+data page is dictionary encoded or uses another encoding. The dictionary page is written first,
+before the data pages of the column chunk.
Review Comment:
The new wording implies that the first N pages of a column chunk can be dictionary-encoded and all other pages after need to be non-dictionary (specifically the "fall back to another
valid encoding for subsequent data pages"). However, IIUC it is valid to interleave non-dict- and dict-encoded pages within a column chunk as long as only one dictionary is used. If you agree, maybe simpler wording could be something like:
> Within a column chunk, the dictionary page is written first, before the data pages of the column chunk. After the dictionary page, data pages can use dictionary encodings or another valid encoding for the column's data type. Dictionary and non-dictionary encoded data pages may be interleaved. Readers must use each page header's `encoding` field to determine how the page should be decoded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-404: Clarify dictionary fallback encoding [parquet-format]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #609:
URL: https://github.com/apache/parquet-format/pull/609#issuecomment-5792586603
The wording looks reasonable to me. But I'm not a native speaker so perhaps @alamb @emkornfield could take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org