You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "PiotrSrebrny (via GitHub)" <gi...@apache.org> on 2025/10/10 09:32:55 UTC
[PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
PiotrSrebrny opened a new pull request, #8586:
URL: https://github.com/apache/arrow-rs/pull/8586
# Which issue does this PR close?
- Closes #8534.
# Are these changes tested?
Changes where not tested and should be covered by existing.
# Are there any user-facing changes?
* added the `ArrowWriter::sync()` function that flushes the underlying writer
* added the `SerializedFileWriter::flush()` function that flushes the underlying writer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#issuecomment-3392558713
I think it is worth modifying the documentation on `ArrowWriter::flush` to make clear that the underlying writer is not flushed, and point to `sync` if such a flush is desired. Can be a followup PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "PiotrSrebrny (via GitHub)" <gi...@apache.org>.
PiotrSrebrny commented on PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#issuecomment-3396228456
I added a comment you suggested. I think it is nice to have it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#issuecomment-3403021004
Thanks @etseidl and @PiotrSrebrny 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#issuecomment-3402144898
Thanks @PiotrSrebrny!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#discussion_r2420773161
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -348,6 +348,11 @@ impl<W: Write + Send> ArrowWriter<W> {
self.writer.write_all(buf)
}
+ /// Flushes underlying writer
+ pub fn sync(&mut self) -> std::io::Result<()> {
Review Comment:
That is a great point. 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "PiotrSrebrny (via GitHub)" <gi...@apache.org>.
PiotrSrebrny commented on PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#issuecomment-3392220379
No hurry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#discussion_r2420297677
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -348,6 +348,11 @@ impl<W: Write + Send> ArrowWriter<W> {
self.writer.write_all(buf)
}
+ /// Flushes underlying writer
+ pub fn sync(&mut self) -> std::io::Result<()> {
Review Comment:
technically speaking you can achieve this with `inner_writer().flush()` but I think this API is nicer and less easy to abuse 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "PiotrSrebrny (via GitHub)" <gi...@apache.org>.
PiotrSrebrny commented on code in PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#discussion_r2420423076
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -348,6 +348,11 @@ impl<W: Write + Send> ArrowWriter<W> {
self.writer.write_all(buf)
}
+ /// Flushes underlying writer
+ pub fn sync(&mut self) -> std::io::Result<()> {
Review Comment:
Hmm, there is no `inner_writer()`. However, `ArrowWriter::inner().flush()` will flush the user provided writer bypassing `BufWriter`, which does not help as data is stucked in `BufWriter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "PiotrSrebrny (via GitHub)" <gi...@apache.org>.
PiotrSrebrny commented on PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#issuecomment-3391923352
Hmm, this is my first PR, is there anything missing here in order to merge this in?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [parquet] Add a sync fn to ArrowWriter that flushes Writer [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8586:
URL: https://github.com/apache/arrow-rs/pull/8586#issuecomment-3392214844
> Hmm, this is my first PR, is there anything missing here in order to merge this in?
Nope, sorry @PiotrSrebrny -- I plan to leave this open for another few days to ensure anyone who is interested has a chance to review it before merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org