You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "1fanwang (via GitHub)" <gi...@apache.org> on 2026/09/18 05:13:07 UTC
[PR] GH-2870: Report compressed dictionary page sizes in the CLI [parquet-java]
1fanwang opened a new pull request, #3798:
URL: https://github.com/apache/parquet-java/pull/3798
### Rationale for this change
`parquet pages` overstates dictionary sizes in compressed files. For the GZIP file below, it reports 248 B even though the stored dictionary is 33 B. That makes the output misleading when investigating compression.
The data-page fix in https://github.com/apache/parquet-java/pull/3326 left the dictionary row using the decompressed size.
Closes https://github.com/apache/parquet-java/issues/2870.
### What changes are included in this PR?
Use the existing page-header lookup for dictionary size and average size, as the data-page rows already do. Files and the core page API are unchanged.
### Are these changes tested?
#### Testing Done
Built the standalone CLI with JDK 17 and Thrift 0.24.0, then ran both versions against the same file:
| Scenario | Command | Before | After |
| --- | --- | --- | --- |
| GZIP dictionary | `java -jar <cli.jar> pages -c color example.parquet` | 248 B; 124.00 B per entry | 33 B; 16.50 B per entry |
| Stored page header | `java -jar <cli.jar> pages --raw -c color example.parquet` | 33 compressed bytes | Unchanged |
The baseline is https://github.com/apache/parquet-java/commit/2df8d02678dab4bb8b926a0d3221cc652984c7ab. The fixed jar comes from this branch. Build commands, with the resulting runtime jars saved as `before-cli.jar` and `after-cli.jar`:
```sh
./mvnw -B -ntp -pl parquet-cli -am -Plocal -DskipTests package
./mvnw -B -ntp -pl parquet-cli -am -Plocal -Dtest=ShowPagesCommandTest,ConvertCSVCommandTest -Dsurefire.failIfNoSpecifiedTests=false package
```
Commands run with those jars:
```sh
python3 -c 'from pathlib import Path; Path("input.csv").write_text("color\n" + ("a" * 120 + "\n" + "b" * 120 + "\n") * 100)'
java -jar before-cli.jar convert-csv input.csv --require color --compression-codec GZIP -o example.parquet
java -jar before-cli.jar pages -c color example.parquet
java -jar after-cli.jar pages -c color example.parquet
java -jar after-cli.jar pages --raw -c color example.parquet
```
<details>
<summary>Raw logs</summary>
```text
Before
Column: color
--------------------------------------------------------------------------------
page type enc count avg size size rows nulls min / max
0-D dict G _ 2 124.00 B 248 B
0-1 data G R 200 0.13 B 25 B
After
Column: color
--------------------------------------------------------------------------------
page type enc count avg size size rows nulls min / max
0-D dict G _ 2 16.50 B 33 B
0-1 data G R 200 0.13 B 25 B
Raw headers
Start of chunk (rowGroup: 0, columnName: color, dictPageOffset: 4, dataPageOffset: 57, numValues: 200, totalSize: 102)
Page 0. (offset: 4, headerSize: 20)
{
"compressed_page_size" : 33,
"crc" : -205207594,
"dictionary_page_header" : {
"encoding" : 2,
"num_values" : 2
},
"type" : 2,
"uncompressed_page_size" : 248
}
Page 1. (offset: 57, headerSize: 24)
{
"compressed_page_size" : 25,
"crc" : 332693654,
"data_page_header" : {
"definition_level_encoding" : 4,
"encoding" : 2,
"num_values" : 200,
"repetition_level_encoding" : 4
},
"type" : 0,
"uncompressed_page_size" : 27
}
End of chunk (offset: 105)
```
</details>
Repeated the CLI comparison with V2 pages and an uncompressed file. The GZIP dictionary reports 33 B in both versions; data-page output and raw headers are unchanged. Uncompressed output is identical before and after.
The regression writes real files with both page versions, GZIP or no compression, and dictionary encoding on or off. It compares the command's size and average-size columns with the original page headers. The GZIP dictionary cases fail before the fix.
### Are there any user-facing changes?
Dictionary rows now report on-disk compressed bytes rather than decompressed bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-2870: Report compressed dictionary page sizes in the CLI [parquet-java]
Posted by "divjotarora (via GitHub)" <gi...@apache.org>.
divjotarora commented on code in PR #3798:
URL: https://github.com/apache/parquet-java/pull/3798#discussion_r4047831279
##########
parquet-cli/src/main/java/org/apache/parquet/cli/commands/ShowPagesCommand.java:
##########
@@ -229,9 +229,7 @@ private long getPageCompressedSize() {
}
private String printDictionaryPage(DictionaryPage dict) {
- // TODO: the compressed size of a dictionary page is lost in Parquet
- dict.getUncompressedSize();
- long totalSize = dict.getCompressedSize();
+ long totalSize = getPageCompressedSize();
Review Comment:
`getPageCompressedSize` calls `columnChunk.hasDictionaryPage()` to determine the page offset. This function does:
```
public boolean hasDictionaryPage() {
EncodingStats stats = getEncodingStats();
if (stats != null) {
// ensure there is a dictionary page and that it is used to encode data pages
return stats.hasDictionaryPages() && stats.hasDictionaryEncodedPages();
}
Set<Encoding> encodings = getEncodings();
return (encodings.contains(PLAIN_DICTIONARY) || encodings.contains(RLE_DICTIONARY));
}
```
In an edge case where a writer emits a dictionary page followed by no `PLAIN_DICTIONARY` or `RLE_DICTIONARY` data pages, we would get wrong results because `hasDictionaryPage()` returns false.
Realistically, writers wouldn't do this, but I didn't find any wording in the spec explicitly disallowing it. The `TestParquetFileWriter` [code](https://github.com/apache/parquet-java/blob/3f1261ce08dc50d02dc8d93d19413b2e0cf7d985/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestParquetFileWriter.java#L229-L235) in this repo seems to do exactly this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-2870: Report compressed dictionary page sizes in the CLI [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3798:
URL: https://github.com/apache/parquet-java/pull/3798#discussion_r4240420541
##########
parquet-cli/src/main/java/org/apache/parquet/cli/commands/ShowPagesCommand.java:
##########
@@ -229,9 +229,7 @@ private long getPageCompressedSize() {
}
private String printDictionaryPage(DictionaryPage dict) {
- // TODO: the compressed size of a dictionary page is lost in Parquet
- dict.getUncompressedSize();
- long totalSize = dict.getCompressedSize();
+ long totalSize = getPageCompressedSize();
Review Comment:
At least this change is not worse than before. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-2870: Report compressed dictionary page sizes in the CLI [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #3798:
URL: https://github.com/apache/parquet-java/pull/3798
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org