You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/08/24 05:57:44 UTC
[PR] [core] Avoid Variant buffer copies in Arrow and Parquet paths [paimon]
JingsongLi opened a new pull request, #9371:
URL: https://github.com/apache/paimon/pull/9371
### Purpose
Avoid materializing Variant metadata and value as byte arrays while moving data through columnar, Arrow, serialization, shredding, and Parquet write paths.
### Changes
- Add mandatory ByteBuffer accessors to Variant and implement a ByteBuffer-backed ColumnarVariant.
- Expose ByteBuffer views from column vectors, including direct ArrowBuf views for Arrow variable-binary vectors.
- Read and write Arrow Variant values without eagerly copying their metadata and value buffers.
- Update Variant readers, builders, serializers, shredding, and row-format writers to consume buffers.
- Reuse writer-owned scratch arrays when Parquet requires a heap-backed Binary, avoiding a large temporary allocation per row.
- Add regression coverage using direct, padded, and buffer-only Variant inputs.
### Verification
- 18 targeted tests passed under the normal Maven profile across paimon-common, paimon-format, and paimon-arrow.
- Local end-to-end Arrow batch to Parquet measurements with approximately 1 MiB Variant values showed 1.24x throughput and 23.8% less allocation without compression. With Zstd, throughput was compression-dominated while allocation decreased by 15.4%.
The local microbenchmark changes are intentionally not included in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reduce Variant buffer copies across Arrow and Parquet paths [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9371:
URL: https://github.com/apache/paimon/pull/9371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reduce Variant buffer copies across Arrow and Parquet paths [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #9371:
URL: https://github.com/apache/paimon/pull/9371#discussion_r3841483534
##########
paimon-arrow/src/main/java/org/apache/paimon/arrow/writer/ArrowFieldWriters.java:
##########
@@ -591,11 +597,14 @@ protected void doWrite(
InternalRow rowData = rowColumnVector.getRow(row);
if (variantSchema != null && rowData.getFieldCount() != fieldCount) {
Review Comment:
This discriminator is ambiguous for a valid two-field top-level shredding schema. For example, produces , while the logical columnar Variant row is ; both have a field count of 2. This branch therefore treats the logical row as an already-shredded row and writes the two buffers in the wrong order. I reproduced the -> round trip, which fails with . Could we distinguish the input representation explicitly (or structurally) instead of by field count, and add a regression test for this two-field schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Reduce Variant buffer copies across Arrow and Parquet paths [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #9371:
URL: https://github.com/apache/paimon/pull/9371#discussion_r3841483534
##########
paimon-arrow/src/main/java/org/apache/paimon/arrow/writer/ArrowFieldWriters.java:
##########
@@ -591,11 +597,14 @@ protected void doWrite(
InternalRow rowData = rowColumnVector.getRow(row);
if (variantSchema != null && rowData.getFieldCount() != fieldCount) {
Review Comment:
This discriminator is ambiguous for a valid two-field top-level shredding schema. For example, variantShreddingSchema(DataTypes.VARIANT()) produces [metadata, value], while the logical columnar Variant row is [value, metadata]; both have a field count of 2. This branch therefore treats the logical row as an already-shredded row and writes the two buffers in the wrong order. I reproduced the ArrowFormatWriter.write(ColumnVector[], ...) -> ArrowBatchReader round trip, which fails with MALFORMED_VARIANT. Could we distinguish the input representation explicitly (or structurally) instead of by field count, and add a regression test for this two-field schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org