You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pchintar (via GitHub)" <gi...@apache.org> on 2026/04/26 21:17:29 UTC
[I] Avoid repetitive heap allocations in IPC writer [arrow-rs]
pchintar opened a new issue, #9835:
URL: https://github.com/apache/arrow-rs/issues/9835
## Description
When writing IPC data using `StreamWriter` or `FileWriter`, the current implementation performs repeated heap allocations and full buffer copies for every record batch, even when writing batches with identical schema and structure.
This leads to unnecessary latency overhead, especially in high-frequency batch writes and streaming pipelines.
---
## Root Cause
Currently in `arrow-ipc/src/writer.rs`, the writer path is structured as:
```text
RecordBatch
→ encode() → EncodedData
→ write_message()
```
The key issue is that `EncodedData` owns its buffers:
```rust
pub struct EncodedData {
pub ipc_message: Vec<u8>,
pub arrow_data: Vec<u8>,
}
```
This forces:
* allocation of new buffers per batch
* copying of flatbuffer data into `Vec<u8>`
* destruction of all intermediate buffers after each write
---
## Current Behavior
For every batch, the following occurs:
```text
1. Build FlatBuffer (fbb)
2. Copy it → ipc_message.to_vec() (Full Copy)
3. Allocate arrow_data Vec
4. Allocate metadata vectors
5. Return EncodedData (owned)
6. write_message() writes data
7. All buffers dropped
```
### Implications
* repeated heap allocations
* repeated memory growth/reallocation
* full flatbuffer copy per batch
* memory churn (alloc → free → alloc)
---
## Proposed Solution
For repeated batch writes, the writer should ideally:
```text
1. Reuse FlatBufferBuilder
2. Reuse arrow_data buffer
3. Reuse metadata vectors
4. Avoid copying flatbuffer data
5. Write directly from existing buffers
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-5014514861
I think #10125 closes this as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-5023467063
I think https://github.com/apache/arrow-rs/pull/10262 just avoids reallocations. It doens't prevent the copying in the first place 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repetitive heap allocations in IPC writer [arrow-rs]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-4323157909
cc @alamb and @etseidl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "Ignalina (via GitHub)" <gi...@apache.org>.
Ignalina commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-4513457017
I have a compression archive format (znippy) that stores already compressed files (like jars/gz etc) directly as a as LargeBinary in Arrow IPC Stream. On a 32-core AMD with pcie 4.0 NVMe:
- Direct write_all() to NVMe: 2775 MB/s
- Arrow IPC StreamWriter: 777 MB/s
The bottleneck is write_buffer() at writer.rs:2083:
arrow_data.extend_from_slice(buffer);
A path that avoids this redundant copy would recover the full 2775 MB/s immediately.
https://github.com/Ignalina/znippy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-4347382594
> > It would help review your PRs quickly if you could break them up into small individual PRs that are easy to review / test -- that typically makes them easier to review and merge
>
> Yea [@alamb](https://github.com/alamb) sure but I previously hesitated to do that bc then it would appear like I'm spamming multiple PRs which actually have the same goal 😅
Yeah, I agree we probably would have been annoyed at a bunch of stacked PRs (especially without evidence of performance speedup).
However, now that we have the evidence I think we'll be able to find sufficient review bandwidth to get them reviewed / merged quickly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "pchintar (via GitHub)" <gi...@apache.org>.
pchintar commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-4330338051
> In general, avoding copies / allocations sounds good to me
>
> It would help review your PRs quickly if you could break them up into small individual PRs that are easy to review / test -- that typically makes them easier to review and merge
>
> Thank you again
Yea @alamb sure but I previously hesitated to do that bc then it would appear like I'm spamming multiple PRs which actually have the same goal 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-4329632021
In general, avoding copies / allocations sounds good to me
It would help review your PRs quickly if you could break them up into small individual PRs that are easy to review / test -- that typically makes them easier to review and merge
Thank you again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-5063436791
> Yes, https://github.com/apache/arrow-rs/pull/10262 avoids intermediate vector re-allocations, but the allocator still needs to call mmap & munmap repeatedly for however many buffers we need, even if we just gave a perfectly sized buffer back.
I am not sure that allocators call `mmap` and `munmap` 🤔 on all allocations (it depends on the allocator and the size of the allocation, as I recall)
I wonder if instead of trying to do the buffer recycling internally, we contemplated adding a new API for the caller of the ipc writer to provide the target buffer (maybe Vec?) that would be checked before allocating a fresh one 🤔 That way it would be opt in behavior and the consumer of the buffer is probably in the best place to reuse it
That being said, finding some realistic benchmark might be hard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-5028700965
```
1. Reuse FlatBufferBuilder
2. Reuse arrow_data buffer
3. Reuse metadata vectors
4. Avoid copying flatbuffer data
5. Write directly from existing buffers
```
1,3,5 were completed in seperate components for #10125 iirc.
The core issue with 2 & 4 is the ownership model. I attempted, on two separate occasions, to implement a buffer pool for the FlightData chunks being sent down to the gRPC layer for transmission over the wire.
the concept being `split_grpc_response` ideally produces equal-sized chunks, so rather than going through a full `allocation`/`deallocation` cycle each time for memory whose size is known ahead of time, we could just reuse a set of perfectly-sized buffers. This mattered even more given that, past a certain threshold (128KB on Linux), memory chunks are allocated and freed via mmap/munmap directly, rather than being retained by the allocator in the process for potential reuse. By default the`target_grpc_max_size` is set to 2MB's.
This was done by creating buffers using [Bytes::from_owner()](https://docs.rs/bytes/latest/bytes/struct.Bytes.html#method.from_owner), so that on drop we could intercept the buffer and place it back into the buffer pool. The exact commit that introduced this is [here](https://github.com/apache/arrow-rs/pull/10137/changes/5ceb08de716c9d3ee5d731b9d6800769e2f28760)
I ran into 4 main issues:
1. Negligible benchmark impact. *(for the most part)* The benchmarks showed very little difference in either direction, no matter how much I tuned max_target_size.
2. We're effectively re-implementing allocator behavior. As mentioned above, up to a certain point, allocators already handle memory pooling better, since so much time and engineering effort has gone into them. We'd need a threshold to decide when pooling turns on, or even gate it behind a cfg.
3. Deallocation started showing up in the profile. Say we have 40MB of data split into 4MB batches to send over the wire, and our buffer pool is working as intended, keeping 5 buffers "hot" so we never need to ask the allocator for more memory. Once the last 4MB chunk is sent over the wire and FlightEncoder is dropped, this triggers one large munmap call that deallocates all 20MB (5 × 4MB) of memory at once.
4. We don't control when buffers are dropped. [FlightEncoder pushes buffers down](https://github.com/apache/arrow-rs/blob/0cc07133c659489c2c03a2ad08994b46910e152f/arrow-flight/src/encode.rs#L407) one FlightData chunk at a time, but we have no way to guarantee when a buffer is "consumed" and subsequently dropped. This compounds issue 3: if the buffer pool isn't effective and we have 10 chunks of data alive at the same time, they'll all return to the pool when dropped — and all get deallocated at once when FlightEncoder itself is dropped.
> @alamb I think https://github.com/apache/arrow-rs/pull/10262 just avoids reallocations. It doens't prevent the copying in the first place 🤔
Yes, to a degree #10262 avoids intermediate vector re-allocations, but the allocator still needs to call `mmap` & `munmap` repeatedly for however many buffers we need, even if we just gave a perfectly sized buffer back.
I hope that make sense, let me know if i missed something
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Avoid repeated heap allocations and buffer copies in IPC writer [arrow-rs]
Posted by "Ignalina (via GitHub)" <gi...@apache.org>.
Ignalina commented on issue #9835:
URL: https://github.com/apache/arrow-rs/issues/9835#issuecomment-5226814970
> > Yes, [#10262](https://github.com/apache/arrow-rs/pull/10262) avoids intermediate vector re-allocations, but the allocator still needs to call mmap & munmap repeatedly for however many buffers we need, even if we just gave a perfectly sized buffer back.
>
> I am not sure that allocators call `mmap` and `munmap` 🤔 on all allocations (it depends on the allocator and the size of the allocation, as I recall)
>
> I wonder if instead of trying to do the buffer recycling internally, we contemplated adding a new API for the caller of the ipc writer to provide the target buffer (maybe Vec?) that would be checked before allocating a fresh one 🤔 That way it would be opt in behavior and the consumer of the buffer is probably in the best place to reuse it
>
> That being said, finding some realistic benchmark might be hard
Ive would be glad to test it out, im guessing a bit but sounds like what been looking for.
>>Direct write_all() to NVMe: 2775 MB/s
>>Arrow IPC StreamWriter: 777 MB/s
A path that avoids this redundant copy would recover the full 2775 MB/s immediately.
https://github.com/Ignalina/znippy
/Rickard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org