You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "V0ldek (via GitHub)" <gi...@apache.org> on 2024/07/26 18:32:11 UTC
[PR] change ipc::reader and writer APIs for consistent buffering [arrow-rs]
V0ldek opened a new pull request, #6132:
URL: https://github.com/apache/arrow-rs/pull/6132
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #6099.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Current writer API automatically wraps the supplied `std::io::Writer` impl into a `BufWriter`.
It is cleaner and more idiomatic to have the default be using the supplied impl directly, as the user might already have a `BufWriter` or an impl that doesn't actually benefit from buffering at all.
StreamReader does a similar thing, but it also exposes a `try_new_unbuffered` that bypasses the internal wrap. These APIs are incosistent between reader/writer and also between `Stream` and `File` versions, but there's no strong reason for that.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
Here we propose a consistent and non-buffered by default API:
- `try_new` does not wrap the passed reader/writer,
- `try_new_buffered` is a convenience function that does wrap the reader/writer into a BufReader/BufWriter,
- all four publicly exposed IPC reader/writers follow the above consistently, i.e. `StreamReader`, `FileReader`, `StreamWriter`, `FileWriter`.
An additional tweak: removed the generic type bounds from struct definitions on the four types, as that is the idiomatic Rust approach (see e.g. stdlib's `HashMap` that has no bounds on the struct definition, only the impl requires `Hash + Eq`).
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
- [BREAKING] function `arrow_ipc::reader::StreamReader<R>::try_new_unbuffered` is removed;
- [BREAKING] function `arrow_ipc::reader::StreamReader<std::io::BufReader<R>>::try_new` is renamed to `try_new_buffered`;
- function `arrow_ipc::reader::FileReader<std::io:::BufReader<R>>::try_new_buffered` is added;
- function `arrow_ipc::writer::StreamWriter<std::io::BufWriter<W>>::try_new_buffered` is added;
- function `arrow_ipc::writer::FileWriter<std::io::BufWriter<W>>::try_new_buffered` is added.
The fact that `try_new` is now unbuffered is a silent change from the API perspective. A number of rustdocs explicitly referenced the internal buffering and have been reworded accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change ipc::reader and writer APIs for consistent buffering [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6132:
URL: https://github.com/apache/arrow-rs/pull/6132#discussion_r1693934161
##########
arrow-ipc/src/reader.rs:
##########
@@ -1043,6 +1043,15 @@ impl<R: Read + Seek> fmt::Debug for FileReader<R> {
}
}
+impl<R: Read + Seek> FileReader<BufReader<R>> {
+ /// Try to create a new file reader with the reader wrapped in a BufReader.
+ ///
+ /// See [`FileReader::try_new`] for an unbuffered version.
Review Comment:
Could you also please add a note to `FileReader::try_new` that it does not do any internal buffering, and if users want buffering they should see `FileReader::try_new_buffered`?
##########
arrow-ipc/src/writer.rs:
##########
@@ -844,6 +844,15 @@ pub struct FileWriter<W: Write> {
data_gen: IpcDataGenerator,
}
+impl<W: Write> FileWriter<BufWriter<W>> {
+ /// Try to create a new file writer with the writer wrapped in a BufWriter.
+ ///
+ /// See [`FileWriter::try_new`] for an unbuffered version.
+ pub fn try_new_buffered(writer: W, schema: &Schema) -> Result<Self, ArrowError> {
+ Self::try_new(BufWriter::new(writer), schema)
+ }
+}
+
impl<W: Write> FileWriter<W> {
/// Try to create a new writer, with the schema written as part of the header
Review Comment:
```suggestion
/// Try to create a new writer, with the schema written as part of the header
///
/// Note the created writer is not buffered. See [`FileWriter::try_new_buffered`] for details.
```
##########
arrow-ipc/src/writer.rs:
##########
@@ -990,16 +998,20 @@ impl<W: Write> FileWriter<W> {
Ok(())
}
- /// Unwraps the BufWriter housed in FileWriter.writer, returning the underlying
- /// writer
+ /// Unwraps the the underlying writer.
+ ///
+ /// The writer is flushed and the FileWriter is finished before returning.
///
- /// The buffer is flushed and the FileWriter is finished before returning the
- /// writer.
+ /// # Errors
+ ///
+ /// An ['Err'](Result::Err) may be returned if an error occurs while finishing the StreamWriter
Review Comment:
👍
##########
arrow-ipc/src/writer.rs:
##########
@@ -853,12 +862,11 @@ impl<W: Write> FileWriter<W> {
/// Try to create a new writer with IpcWriteOptions
Review Comment:
```suggestion
/// Try to create a new writer with IpcWriteOptions
///
/// Note the created writer is not buffered. See [`FileWriter::try_new_buffered`] for details.
```
##########
arrow-ipc/src/reader.rs:
##########
@@ -1163,21 +1172,24 @@ impl<R: Read> fmt::Debug for StreamReader<R> {
}
impl<R: Read> StreamReader<BufReader<R>> {
- /// Try to create a new stream reader with the reader wrapped in a BufReader
+ /// Try to create a new stream reader with the reader wrapped in a BufReader.
///
- /// The first message in the stream is the schema, the reader will fail if it does not
- /// encounter a schema.
- /// To check if the reader is done, use `is_finished(self)`
- pub fn try_new(reader: R, projection: Option<Vec<usize>>) -> Result<Self, ArrowError> {
- Self::try_new_unbuffered(BufReader::new(reader), projection)
+ /// See [`StreamReader::try_new`] for an unbuffered version.
+ pub fn try_new_buffered(reader: R, projection: Option<Vec<usize>>) -> Result<Self, ArrowError> {
+ Self::try_new(BufReader::new(reader), projection)
}
}
impl<R: Read> StreamReader<R> {
- /// Try to create a new stream reader but do not wrap the reader in a BufReader.
+ /// Try to create a new stream reader.
+ ///
+ /// The first message in the stream is the schema, the reader will fail if it does not
+ /// encounter a schema.
+ /// To check if the reader is done, use [`is_finished(self)`](StreamReader::is_finished).
///
- /// Unless you need the StreamReader to be unbuffered you likely want to use `StreamReader::try_new` instead.
- pub fn try_new_unbuffered(
Review Comment:
To ease migration, what do you think about adding a `try_new_unbuffered` function (that calls `try_new`) marked as `deprecated` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change ipc::reader and writer APIs for consistent buffering [arrow-rs]
Posted by "V0ldek (via GitHub)" <gi...@apache.org>.
V0ldek commented on code in PR #6132:
URL: https://github.com/apache/arrow-rs/pull/6132#discussion_r1693990762
##########
arrow-ipc/src/reader.rs:
##########
@@ -1163,21 +1172,24 @@ impl<R: Read> fmt::Debug for StreamReader<R> {
}
impl<R: Read> StreamReader<BufReader<R>> {
- /// Try to create a new stream reader with the reader wrapped in a BufReader
+ /// Try to create a new stream reader with the reader wrapped in a BufReader.
///
- /// The first message in the stream is the schema, the reader will fail if it does not
- /// encounter a schema.
- /// To check if the reader is done, use `is_finished(self)`
- pub fn try_new(reader: R, projection: Option<Vec<usize>>) -> Result<Self, ArrowError> {
- Self::try_new_unbuffered(BufReader::new(reader), projection)
+ /// See [`StreamReader::try_new`] for an unbuffered version.
+ pub fn try_new_buffered(reader: R, projection: Option<Vec<usize>>) -> Result<Self, ArrowError> {
+ Self::try_new(BufReader::new(reader), projection)
}
}
impl<R: Read> StreamReader<R> {
- /// Try to create a new stream reader but do not wrap the reader in a BufReader.
+ /// Try to create a new stream reader.
+ ///
+ /// The first message in the stream is the schema, the reader will fail if it does not
+ /// encounter a schema.
+ /// To check if the reader is done, use [`is_finished(self)`](StreamReader::is_finished).
///
- /// Unless you need the StreamReader to be unbuffered you likely want to use `StreamReader::try_new` instead.
- pub fn try_new_unbuffered(
Review Comment:
Definitely no harm in that :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove automatic buffering in `ipc::reader::FileReader` for for consistent buffering [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6132:
URL: https://github.com/apache/arrow-rs/pull/6132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change ipc::reader and writer APIs for consistent buffering [arrow-rs]
Posted by "V0ldek (via GitHub)" <gi...@apache.org>.
V0ldek commented on PR #6132:
URL: https://github.com/apache/arrow-rs/pull/6132#issuecomment-2253299892
This is a breaking change and should be labelled with `api-change`, but I can't add it as an external contributor :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove automatic buffering in `ipc::reader::FileReader` for for consistent buffering [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #6132:
URL: https://github.com/apache/arrow-rs/pull/6132#discussion_r1694220430
##########
arrow-ipc/src/reader.rs:
##########
@@ -1224,6 +1246,15 @@ impl<R: Read> StreamReader<R> {
})
}
+ /// Deprecated, use [`StreamReader::try_new`] instead.
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] change ipc::reader and writer APIs for consistent buffering [arrow-rs]
Posted by "V0ldek (via GitHub)" <gi...@apache.org>.
V0ldek commented on code in PR #6132:
URL: https://github.com/apache/arrow-rs/pull/6132#discussion_r1693982396
##########
arrow-ipc/src/writer.rs:
##########
@@ -990,16 +998,20 @@ impl<W: Write> FileWriter<W> {
Ok(())
}
- /// Unwraps the BufWriter housed in FileWriter.writer, returning the underlying
- /// writer
+ /// Unwraps the the underlying writer.
+ ///
+ /// The writer is flushed and the FileWriter is finished before returning.
///
- /// The buffer is flushed and the FileWriter is finished before returning the
- /// writer.
+ /// # Errors
+ ///
+ /// An ['Err'](Result::Err) may be returned if an error occurs while finishing the StreamWriter
Review Comment:
I'm going to add those Errors sections to the `try_new*` functions as well in hindisght.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org