You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pchintar (via GitHub)" <gi...@apache.org> on 2026/04/18 21:27:31 UTC

[PR] feat(ipc): Remove per-message flush in IPC writer hot path [arrow-rs]

pchintar opened a new pull request, #9763:
URL: https://github.com/apache/arrow-rs/pull/9763

   # Which issue does this PR close?
   
   - Closes #9762 .
   
   # Rationale for this change
   
   Currently, `flush()` is called unconditionally in `write_body_buffers` and `write_continuation` - both executed per batch. This forces per-batch syscalls, breaks write coalescing, and adds unnecessary overhead. These flushes are not required for correctness (IPC boundaries are length-prefixed, no durability guarantees).
   
   # What changes are included in this PR?
   
   - Remove `flush()` from `write_body_buffers` and `write_continuation`
   - Add `self.writer.flush()?` to `StreamWriter::finish()` (missing; `FileWriter::finish()` already has it)
   
   # Are these changes tested?
   
   Yes the changes were tested successfully by running:
   ```bash
   cargo test -p arrow-ipc
   cargo fmt --all
   cargo clippy --all-targets --all-features -- -D warnings
   ```
   
   # Are there any user-facing changes?
   
   No, There are no changes made to any Public APIs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(ipc): Remove per-message flush in IPC writer hot path [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9763:
URL: https://github.com/apache/arrow-rs/pull/9763


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(ipc): Remove per-message flush in IPC writer hot path [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9763:
URL: https://github.com/apache/arrow-rs/pull/9763#discussion_r3106897875


##########
arrow-ipc/src/writer.rs:
##########
@@ -1467,6 +1467,7 @@ impl<W: Write> StreamWriter<W> {
         }
 
         write_continuation(&mut self.writer, &self.write_options, 0)?;
+        self.writer.flush()?;

Review Comment:
   It makes sense to me to flush the stream on completion



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] feat(ipc): Remove per-message flush in IPC writer hot path [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9763:
URL: https://github.com/apache/arrow-rs/pull/9763#issuecomment-4290982137

   Thanks again @pchintar 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org