You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "discivigour (via GitHub)" <gi...@apache.org> on 2026/09/17 11:48:41 UTC

[PR] [format] Use buffered encoder for Avro writes [paimon]

discivigour opened a new pull request, #9928:
URL: https://github.com/apache/paimon/pull/9928

   ### Purpose
   
   Use Avro's `BufferedBinaryEncoder` for data-file records so small field encodings are batched before they enter the Avro block buffer. This improves nested MAP/ARRAY writes and avoids a regression for common scalar schemas. Add coverage for closing with a partially filled encoder buffer and for mixing buffered records with raw block copies across the supported codecs.
   
   ### Tests
   
   - `mvn -pl paimon-format -am clean test -Dmaven.repo.local=/opt/homebrew/opt/maven/repository -DskipITs -Dcheckstyle.skip=true -Dtest=AvroFileFormatTest -Dsurefire.failIfNoSpecifiedTests=false` (37 tests passed)
   
   A/B benchmarks ran on the same 4-core ECS/JDK 11 process against `fx_perf_rest.codex_db`; each mode used identical inputs and output settings, and all rows/fields, file layouts, block distributions, and decompressed payload CRCs matched.
   
   | Workload | Data per sample | Buffered mean | Direct mean | Result |
   | --- | ---: | ---: | ---: | --- |
   | 20 `MAP<STRING, ARRAY<INT>>` fields; 20 keys/map; 300 ints/key | 5 GiB | 63.362 s | 73.479 s | Buffered 13.77% faster, 3/3 |
   | 20 `BYTES` fields; 512 B/field | 3 GiB | 33.873 s | 33.515 s | Direct 1.06% faster in write stage, 2/3; direction was not stable |
   | 36 scalar fields; 6 each of `INT`, `FLOAT`, `BOOLEAN`, `STRING`, `DECIMAL(18,4)`, `DATE` | 5 GiB | 77.142 s | 78.710 s | Buffered 1.99% faster, 3/3 |
   
   For the BYTES workload, including commit time was 34.036 s for Buffered and 34.360 s for Direct, so it did not demonstrate a stable end-to-end Direct advantage. Results are limited to these three-run workloads and this ECS/OSS environment.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Use buffered encoder for Avro writes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9928:
URL: https://github.com/apache/paimon/pull/9928


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Use buffered encoder for Avro writes [paimon]

Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on PR #9928:
URL: https://github.com/apache/paimon/pull/9928#issuecomment-5724771032

   > Verified the Avro-side safety of this against avro-1.11.4 internals (disassembled `DataFileWriter`/`BufferedBinaryEncoder`), since the risk here is a buffered `bufOut` desynchronizing the block buffer. All three append paths are flush-safe with the buffered encoder:
   > 
   > * `append()` rollback uses `resetBufferTo(bufferInUse)`, which flushes `bufOut` before truncating — so a failed append can't leave partial encoder bytes that survive into the next record. `testBufferedRecordSerializationFailureRollsBack` pins exactly this, including the partial-encoder-buffer case (small payload) vs. flushed case (large payload).
   > * `appendEncoded()` goes through `bufOut.writeFixed`, and `writeBlock()` flushes `bufOut` before the raw block copy, so `appendAllFrom` can't interleave encoder residue with a copied block. `testRecordsAroundRawBlockCopy` (rows + encoded records + raw block copy, across all four codecs) covers the ordering.
   > * `setEncoder` is applied at `create()`, before any append — timing is fine.
   > 
   > Keeping the direct encoder for manifests via `context.isManifest()` is the right call: the buffered encoder copies array-backed `ByteBuffer`s into a temp array on every `appendEncoded`, which is the manifest hot path. And excluding manifests from `FILE_BLOCK_SIZE` (sync interval) while data files get it is consistent with the existing `manifestFormat` behavior.
   > 
   > `testCloseFlushesBufferedRecords` covering all codecs is appreciated — the "leave both a partial encoder buffer and a partial block, then close" setup is exactly the state that would silently truncate.
   
   Thanks for the detailed review and for checking the Avro 1.11.4 internals! I agree that keeping Direct for manifests avoids the extra allocations while retaining Buffered for data-file writes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Use buffered encoder for Avro writes [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #9928:
URL: https://github.com/apache/paimon/pull/9928#discussion_r4043678538


##########
paimon-format/src/main/java/org/apache/paimon/format/avro/AvroFileFormat.java:
##########
@@ -67,13 +68,15 @@ public class AvroFileFormat extends FileFormat {
     private final Options options;
     private final int zstdLevel;
     @Nullable private final MemorySize blockSize;
+    private final boolean useBufferedEncoder;
 
     public AvroFileFormat(FormatContext context) {
         super(IDENTIFIER);
 
         this.options = getIdentifierPrefixOptions(context.options());
         this.zstdLevel = context.zstdLevel();
         this.blockSize = context.blockSize();
+        this.useBufferedEncoder = !context.isManifest();

Review Comment:
   Why disable this for manifest?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [format] Use buffered encoder for Avro writes [paimon]

Posted by "discivigour (via GitHub)" <gi...@apache.org>.
discivigour commented on code in PR #9928:
URL: https://github.com/apache/paimon/pull/9928#discussion_r4043940342


##########
paimon-format/src/main/java/org/apache/paimon/format/avro/AvroFileFormat.java:
##########
@@ -67,13 +68,15 @@ public class AvroFileFormat extends FileFormat {
     private final Options options;
     private final int zstdLevel;
     @Nullable private final MemorySize blockSize;
+    private final boolean useBufferedEncoder;
 
     public AvroFileFormat(FormatContext context) {
         super(IDENTIFIER);
 
         this.options = getIdentifierPrefixOptions(context.options());
         this.zstdLevel = context.zstdLevel();
         this.blockSize = context.blockSize();
+        this.useBufferedEncoder = !context.isManifest();

Review Comment:
   Manifest entries can currently be written through three paths:
   - Normal record serialization via append().
   - Pre-encoded record reuse via appendEncoded().
   - Raw block reuse via appendAllFrom().
   These paths can be mixed within the same writer. I kept the direct encoder for manifests for two reasons:
   
   1. **The pre-encoded record path incurs extra overhead, while raw block copying does not directly benefit.** In Avro 1.11.4, the buffered encoder turns the array-backed `ByteBuffer` passed to `appendEncoded()` into a read-only view, resulting in a temporary array allocation and a full-record copy. Raw block copying bypasses per-record encoding, so buffering provides no direct benefit there.
   
   2. The normal record-writing path may benefit, but we expect the gain to be more limited. In our benchmark with 20 MAP<STRING, ARRAY<INT>> fields, 20 keys per map, and 300 integers per key—120,000 small integer elements per row—the observed improvement was only about 10%. For a 500-column table, manifest entries have far fewer individually encoded small values: min/max statistics are written as byte blobs, with per-column scalar encoding mainly coming from arrays such as nullCounts. This suggests less opportunity for small-write batching. And *enabling it without penalizing the pre-encoded path requires additional implementation code and testing.** It would need mixed-write, rollback, block-boundary, and performance coverage.
   
   For this PR, keeping the existing direct encoder for manifests avoids introducing that overhead and keeps the change scoped to data-file writes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org