You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2025/06/19 09:17:45 UTC
[PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
zhuqi-lucas opened a new pull request, #7714:
URL: https://github.com/apache/arrow-rs/pull/7714
# Which issue does this PR close?
Currently, no pub api to support write the internal buffer for SerializedFileWriter, it's very helpful when we want to add low level API for example:
https://github.com/apache/datafusion/pull/16395
Because that we want to update the buf bytes written, if we use the buf internal file to write, we can't update the internal buf written bytes.
# Rationale for this change
Add API to support write with buf byteswritten updating.
# What changes are included in this PR?
Add API to support write with buf byteswritten updating.
# Are there any user-facing changes?
No
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#issuecomment-2991796048
I would really like to make sure this one is in #7394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#discussion_r2159114019
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
Review Comment:
I was trying to say that using `inner()` is safe because it is immutable (not `&mut`) and thus a caller can't use `inner()` to *write* to the underlying stream (only to read)
I agree that the warning added to `inner_mut` is totally useful and valid. I just think it might be confusing if the same warning is on `inner` as well. Or perhaps I am missing something
> The low level we called by inner(), it has write_all api:
I thought the example that failed called `inner_mut()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#issuecomment-2988393576
Thank you @etseidl for review, i also add the reminder docs in latest PR for users who want to write directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#discussion_r2158897830
##########
parquet/src/file/writer.rs:
##########
@@ -390,13 +390,33 @@ impl<W: Write + Send> SerializedFileWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
Review Comment:
Likewise here since it is not possible to write to an immutable buffer we can probably skip this warning
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
Review Comment:
Given this is a read only reference (immutable) the warning here is probably not necessary.
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
pub fn inner(&self) -> &W {
self.writer.inner()
}
/// Returns a mutable reference to the underlying writer.
///
- /// It is inadvisable to directly write to the underlying writer, doing so
- /// will likely result in a corrupt parquet file
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
pub fn inner_mut(&mut self) -> &mut W {
self.writer.inner_mut()
}
Review Comment:
I wonder should we add `write_all` to the ArrowWriter too?
##########
parquet/src/file/writer.rs:
##########
@@ -390,13 +390,33 @@ impl<W: Write + Send> SerializedFileWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
+ ///
+ /// If you want to write safely to the underlying writer, use [`Self::write_all`].
pub fn inner(&self) -> &W {
self.buf.inner()
}
+ /// Writes the given buf bytes to the internal buffer.
Review Comment:
It might be nice to add a note here about why you might do this
For example, something like
```suggestion
/// Writes the given buf bytes to the internal buffer.
///
/// This can be used to write raw data to an in-progress parquet file, for
/// example custom index structures or other payloads. Other parquet readers
/// will skip this data when reading the files.
```
##########
parquet/src/file/writer.rs:
##########
@@ -390,13 +390,33 @@ impl<W: Write + Send> SerializedFileWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
+ ///
+ /// If you want to write safely to the underlying writer, use [`Self::write_all`].
pub fn inner(&self) -> &W {
self.buf.inner()
}
+ /// Writes the given buf bytes to the internal buffer.
+ ///
+ /// It's safe to use this method to write data to the underlying writer,
+ /// because it will ensure that the buffering and byte‐counting layers are used.
+ pub fn write_all(&mut self, buf: &[u8]) -> std::io::Result<()> {
+ self.buf.write_all(buf)
+ }
+
/// Returns a mutable reference to the underlying writer.
///
- /// It is inadvisable to directly write to the underlying writer.
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
+ ///
+ /// If you want to write safely to the underlying writer, use [`Self::write_all`].
Review Comment:
❤️
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
pub fn inner(&self) -> &W {
self.writer.inner()
}
/// Returns a mutable reference to the underlying writer.
///
- /// It is inadvisable to directly write to the underlying writer, doing so
- /// will likely result in a corrupt parquet file
+ /// **Warning**: if you write directly to this writer, you will skip
Review Comment:
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#discussion_r2159289884
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
Review Comment:
Got it now, thanks @alamb , addressed in latest PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#discussion_r2159046094
##########
parquet/src/file/writer.rs:
##########
@@ -390,13 +390,33 @@ impl<W: Write + Send> SerializedFileWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
+ ///
+ /// If you want to write safely to the underlying writer, use [`Self::write_all`].
pub fn inner(&self) -> &W {
self.buf.inner()
}
+ /// Writes the given buf bytes to the internal buffer.
Review Comment:
Thank you @alamb good suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#issuecomment-2991722295
> Thank you @zhuqi-lucas and @etseidl
>
> I have some more comments about comments 😆 but the code looks great here.
>
> Thank you!
Thank you @alamb for review, addressed comments and suggestions in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#issuecomment-2991828364
> * I would really like to make sure this one is in [Release arrow-rs / parquet Minor version `55.2.0` (June 2025) #7394](https://github.com/apache/arrow-rs/issues/7394)
Sounds great to include in this release!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#discussion_r2159048868
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
pub fn inner(&self) -> &W {
self.writer.inner()
}
/// Returns a mutable reference to the underlying writer.
///
- /// It is inadvisable to directly write to the underlying writer, doing so
- /// will likely result in a corrupt parquet file
+ /// **Warning**: if you write directly to this writer, you will skip
+ /// the `TrackedWrite` buffering and byte‐counting layers. That’ll cause
+ /// the file footer’s recorded offsets and sizes to diverge from reality,
+ /// resulting in an unreadable or corrupted Parquet file.
pub fn inner_mut(&mut self) -> &mut W {
self.writer.inner_mut()
}
Review Comment:
Thank you @alamb , good suggestion!
Addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#discussion_r2159029611
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
Review Comment:
Thank you @alamb for review, because the failed example use the inner() to call write_all(), so i also add warning here, here is the failed example:
https://github.com/apache/datafusion/pull/16395#discussion_r2148133565
The low level we called by inner(), it has write_all api:
```rust
impl<W: Write> BufWriter<W> {
#[inline]
fn write_all(&mut self, buf: &[u8]) -> io::Result<()> {
// Use < instead of <= to avoid a needless trip through the buffer in some cases.
// See `write_all_cold` for details.
if buf.len() < self.spare_capacity() {
// SAFETY: safe by above conditional.
unsafe {
self.write_to_buffer_unchecked(buf);
}
Ok(())
} else {
self.write_all_cold(buf)
}
}
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support write to buffer api for SerializedFileWriter [arrow-rs]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #7714:
URL: https://github.com/apache/arrow-rs/pull/7714#discussion_r2159029611
##########
parquet/src/arrow/arrow_writer/mod.rs:
##########
@@ -320,14 +320,21 @@ impl<W: Write + Send> ArrowWriter<W> {
}
/// Returns a reference to the underlying writer.
+ ///
+ /// **Warning**: if you write directly to this writer, you will skip
Review Comment:
Thank you @alamb for review, because the failed example use the inner() to call write_all(), so i also add warning here, here is the failed example:
https://github.com/apache/datafusion/pull/16395#discussion_r2148133565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org