You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "qzyu999 (via GitHub)" <gi...@apache.org> on 2026/08/21 19:59:09 UTC
[PR] Add page-level ArrowParquetWriter for writing VectorSchemaRoot to Parquet [parquet-java]
qzyu999 opened a new pull request, #3734:
URL: https://github.com/apache/parquet-java/pull/3734
## Summary
Adds `ArrowParquetWriter` to the `parquet-arrow` module — a page-level writer that accepts Arrow `VectorSchemaRoot` batches and produces valid Parquet files without per-row object construction.
Closes #3733. Related: #2264, #3353.
## Design
Bypasses `RecordConsumer`/`ColumnWriter` entirely. Writes assembled pages directly to `PageWriter` with per-column strategy selection:
- **ZeroCopyPlainWriter** (required fixed-width): wraps Arrow data buffer as page `BytesInput` directly. O(1) level encoding, O(N) stats scan, zero per-value dispatch.
- **NullablePlainWriter** (optional fixed-width): single-pass compaction of non-null values with inline statistics.
- **VarWidthPlainWriter** (STRING/BINARY): single-pass transformation from Arrow offset+data buffers to Parquet length-prefixed format.
- **BooleanPlainWriter** (BOOLEAN): bit-pack compaction with null handling.
Does not extend `ParquetWriter<T>` because `ParquetWriter.write(T)` increments its internal record count by 1 per call, which is incompatible with batch semantics.
## Key properties
- Page format: V1 with length-prefixed RLE-encoded RL/DL sections
- Buffer safety: `PageWriter.writePage()` copies bytes immediately via `ConcatenatingByteBufferCollector.collect()` — Arrow buffers can be freed after `writeBatch()` returns
- Thread-safe statistics: `StatsResult` return object, no shared mutable state
- NaN tracking for float/double columns
- Page-size chunking: large batches split into ~1MB pages for column-index effectiveness
- Row group flush: based on actual `PageWriter.getMemSize()`, not heuristic
- Defensive null-buffer checks on all nullable writers
## Types supported
INT32, INT64, FLOAT, DOUBLE, BOOLEAN, BINARY (string), FIXED_LEN_BYTE_ARRAY. Nullable and required. Covers common streaming workload schemas.
## Not included (follow-up PRs)
- Dictionary encoding (requires two-pass algorithm — build dict, then write indices)
- Nested types (requires recursive RL/DL computation)
Both throw clear `UnsupportedOperationException`.
## Tests
10 round-trip tests (write via ArrowParquetWriter, read via standard ParquetReader):
- Required and nullable integers
- Floats/doubles including NaN
- Strings with nulls and empty values
- Booleans with nulls
- Mixed-type schema
- Multi-batch writes
- Large batch page splitting (500K rows)
- Footer row count verification
- Zero-copy path with manually-constructed REQUIRED schema
## Dependencies added to parquet-arrow POM
- `parquet-hadoop` (compile) — for ParquetFileWriter, ColumnChunkPageWriteStore
- `arrow-memory-netty` (test) — Arrow allocator runtime
- `hadoop-common` (test) — for ParquetReader in round-trip tests
- `hadoop-mapreduce-client-core` (test) — required by ParquetReader
- `parquet-hadoop` test-jar (test) — GroupReadSupport
## Discussion point
Adding `parquet-hadoop` as a compile dependency to `parquet-arrow` changes the module's weight. Alternative: create a new `parquet-arrow-hadoop` module. Open to guidance from maintainers.
## How to run tests
```
mvn test -pl parquet-arrow \
-Dsurefire.argLine="--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add page-level ArrowParquetWriter for writing VectorSchemaRoot to Parquet [parquet-java]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3734:
URL: https://github.com/apache/parquet-java/pull/3734#issuecomment-6304597759
This pull request has been automatically marked as stale because it has had no activity for at least 2 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Parquet!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add page-level ArrowParquetWriter for writing VectorSchemaRoot to Parquet [parquet-java]
Posted by "mikepapadim (via GitHub) via issues" <is...@parquet.apache.org>.
mikepapadim commented on PR #3734:
URL: https://github.com/apache/parquet-java/pull/3734#issuecomment-6880604268
I ran this PR as a writer for Apache Fluss lake tiering, which is one of the downstream users named in the description. I used head 1856d5b8, built against parquet 1.18.2 on JDK 21, and the 10 tests pass. The data was a TPC-H lineitem-like table: 19 nullable columns (INT32, INT64, DOUBLE, DATE, STRING, TIMESTAMP) in 16384-row batches, written with ZSTD. For comparison I wrote the same rows with `ExampleParquetWriter`.
**Page sizing.** With dictionary off on both sides, this PR's file is 26.8 MB against 21.7 MB from `ExampleParquetWriter` (128 MB of Arrow input). The gap is in columns where encoding doesn't matter: a date column is 1.94 vs 0.70 MB, and a constant INT32 column is 30.7 vs 2.7 KB. The cause is in `writeBatch`:
```java
int typeLen = col.getPrimitiveType().getTypeLength();
estimatedRowBytes += typeLen > 0 ? typeLen : VAR_WIDTH_ESTIMATE_BYTES;
...
int rowsPerPage = Math.max(1, TARGET_PAGE_SIZE_BYTES / Math.max(estimatedRowBytes, 1));
```
`getTypeLength()` is 0 for every type except FIXED_LEN_BYTE_ARRAY, so each column is estimated at 32 bytes. Because the estimate is summed over all columns, the 1 MB target is spread across the whole row. With 19 columns that gives 1724 rows a page: about 7 KB INT32 pages and 590 pages a column chunk, where `ExampleParquetWriter` writes 49. Each page is compressed on its own, so ZSTD gets very little context.
I tried sizing by the widest column instead, using primitive widths (BOOLEAN 1, INT32/FLOAT 4, INT64/DOUBLE 8, INT96 12, FLBA its length, binary 32), and taking the max instead of the sum:
| Arrow input | as is | per-column page sizing |
|---:|---:|---:|
| 128 MB | 700 ms, 26.8 MB | 457 ms, 21.8 MB |
| 512 MB | 2568 ms, 107.5 MB | 1733 ms, 87.5 MB |
With the change, the output matches `ExampleParquetWriter` with dictionary off (21.79 vs 21.74 MB), and the write is about a third faster. Pages are still capped at one batch, because pages don't span `writeBatch` calls. A real fix should probably also estimate each column's bytes separately (or cut pages per column by bytes), so that wide string columns and narrow INT32 columns both get sensible pages. I have this as a small commit with two tests: `testPageSizeDoesNotShrinkWithColumnCount`, plus an assertion on the page count in `testLargeBatchProducesMultiplePages`, whose comment expects about 2 pages but gets 16 today. It's up as a PR against your branch: https://github.com/qzyu999/parquet-java/pull/1
**Dictionary.** After that change, the remaining gap to a dictionary-encoded file (16.9 MB from `ExampleParquetWriter`) comes from low-cardinality strings and repeated doubles. For example, a 4-value string column is 798 KB plain vs 30 KB with dictionary. So dictionary encoding, which is already listed under follow-ups, matters for size.
For reference, on the same machine (i9-13900K), the row-object path through Iceberg's `GenericRecord` writer takes 7.3 s for the 512 MB.
**JDK note.** On JDK 17+, the writer needs `--add-opens=java.base/java.nio=ALL-UNNAMED`, because `ArrowBuf.nioBuffer()` goes through `MemoryUtil.directBuffer`. It might be worth documenting that next to the writer.
**Formatting.** Running `mvn spotless:apply` in `parquet-arrow` reformats 8 of the new main files and the test. CI hasn't run on this PR yet, so that will probably be the first failure once the workflows are approved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add page-level ArrowParquetWriter for writing VectorSchemaRoot to Parquet [parquet-java]
Posted by "qzyu999 (via GitHub) via issues" <is...@parquet.apache.org>.
qzyu999 commented on PR #3734:
URL: https://github.com/apache/parquet-java/pull/3734#issuecomment-6881628623
Thanks for taking a look and fixing the bug @mikepapadim, merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Add page-level ArrowParquetWriter for writing VectorSchemaRoot to Parquet [parquet-java]
Posted by "qzyu999 (via GitHub) via issues" <is...@parquet.apache.org>.
qzyu999 commented on PR #3734:
URL: https://github.com/apache/parquet-java/pull/3734#issuecomment-6859512322
bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org