[jira] [Created] (SPARK-59599) Pre-size the ArrowConverters.serializeBatch output buffer to avoid repeated reallocation

["David Mollitor (Jira)" <ji...@apache.org>]

David Mollitor created SPARK-59599:
--------------------------------------

             Summary: Pre-size the ArrowConverters.serializeBatch output buffer to avoid repeated reallocation
                 Key: SPARK-59599
                 URL: https://issues.apache.org/jira/browse/SPARK-59599
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.1.0
            Reporter: David Mollitor


h3. Problem

{{ArrowConverters.serializeBatch}} serializes a single {{ArrowRecordBatch}} into a {{byte[]}} through a {{ByteArrayOutputStream}} created with no initial capacity:
{code:scala}
val out = new ByteArrayOutputStream()
val writeChannel = new WriteChannel(Channels.newChannel(out))
MessageSerializer.serialize(writeChannel, batch)
out.toByteArray
{code}
A no-arg {{ByteArrayOutputStream}} starts at the JDK default of 32 bytes and grows by repeatedly doubling its backing array ({{{}grow{}}} + {{{}Arrays.copyOf{}}}).

The batch is written {*}incrementally{*}, not as one block: {{MessageSerializer.serialize}} writes the message metadata and then each of the batch's buffers (with alignment padding) individually, and the {{WritableByteChannel}} returned by {{Channels.newChannel}} further splits every write into chunks of at most 8 KB. So a 32-byte buffer reallocates and copies its contents on the order of {{log2(batchSize / 32)}} times per batch, producing that many short-lived intermediate arrays (GC pressure) plus the copies.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org