[jira] [Created] (AVRO-4240) Size DataFileWriter output buffer to fit entire block frame to reduce write syscalls

["David Mollitor (Jira)" <ji...@apache.org>]

David Mollitor created AVRO-4240:
------------------------------------

             Summary: Size DataFileWriter output buffer to fit entire block frame to reduce write syscalls
                 Key: AVRO-4240
                 URL: https://issues.apache.org/jira/browse/AVRO-4240
             Project: Apache Avro
          Issue Type: Improvement
          Components: java
    Affects Versions: 1.12.1, 1.11.5, 1.10.2
            Reporter: David Mollitor
            Assignee: David Mollitor
             Fix For: 1.13.0


DataFileStream.DataBlock#writeBlockTo writes four pieces sequentially through a DirectBinaryEncoder into a BufferedFileOutputStream that uses the default 8KB BufferedOutputStream buffer:
 # Entry count (varint-encoded long, 1-10 bytes)
 # Block size (varint-encoded long, 1-10 bytes)
 # Compressed block data (~64KB at the default sync interval)
 # Sync marker (16 bytes)

The default sync interval was increased from 16KB to 64KB in AVRO-1398 but the BufferedFileOutputStream buffer size was never adjusted. Since the block data far exceeds the 8KB buffer, BufferedOutputStream flushes the buffered entry count and block size bytes, then writes the block data directly, then the sync marker goes into the buffer and gets flushed again at the end, resulting in at least 3 write syscalls per block instead of 1.

This change sizes the BufferedFileOutputStream buffer to maxBlockSize() + 20 + sync.length so that a complete block frame fits in a single buffer, accumulates all writes, and flushes once.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)