You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "yugan95 (via GitHub)" <gi...@apache.org> on 2026/05/25 09:30:16 UTC
[PR] [format] Pass through parquet statistics and page-size-check config in RowDataParquetBuilder [paimon]
yugan95 opened a new pull request, #7956:
URL: https://github.com/apache/paimon/pull/7956
### Purpose
Pass through parquet statistics and page-size-check configuration in
`RowDataParquetBuilder`.
Currently `RowDataParquetBuilder` does not forward the following Parquet config keys to
the writer:
- `parquet.statistics.truncate.length`
- `parquet.columnindex.truncate.length`
- `parquet.page.size.row.check.min`
- `parquet.page.size.row.check.max`
Without these, users cannot tune Parquet page-size checking behavior or control the
truncation length of statistics and column indexes. This is especially relevant for
tables with large records, where the default page-size check thresholds can lead to
oversized pages.
Split out from #7621 per reviewer feedback, as this is an independent enhancement.
#### Changes
- **`RowDataParquetBuilder`**: add `.withMinRowCountForPageSizeCheck()`,
`.withMaxRowCountForPageSizeCheck()`, `.withStatisticsTruncateLength()`,
`.withColumnIndexTruncateLength()` to the builder chain, reading from the existing
Hadoop `Configuration` with Parquet's default values as fallback.
### Tests
Existing Parquet write tests cover the default config path. The new keys follow the same
pattern as other builder options (e.g. `withDictionaryPageSize`, `withPageSize`) and
use Parquet's built-in defaults when not set.
### API and Format
N/A — no public API or format changes. Users can now set these keys via table
properties, using the same mechanism as other Parquet config keys.
### Documentation
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [format] Pass through parquet statistics and page-size-check config in RowDataParquetBuilder [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #7956:
URL: https://github.com/apache/paimon/pull/7956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [format] Pass through parquet statistics and page-size-check config in RowDataParquetBuilder [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #7956:
URL: https://github.com/apache/paimon/pull/7956#issuecomment-4533755963
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org