You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "kylebarron (via GitHub)" <gi...@apache.org> on 2026/09/21 20:31:20 UTC
[PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
kylebarron opened a new pull request, #11161:
URL: https://github.com/apache/arrow-rs/pull/11161
# Which issue does this PR close?
- Closes https://github.com/apache/arrow-rs/issues/6586.
# Rationale for this change
Arrow-rs doesn't currently make it possible to describe sequences of _arrays_. It only really defines sequences of _Record batches_.
This makes it hard to interface with libraries that use "chunked arrays" concepts or similar.
This PR introduces code that I've used in [`pyo3-arrow`](https://crates.io/crates/pyo3-arrow) for a while. (It was necessary to support interop with a pyarrow `ChunkedArray`).
# What changes are included in this PR?
- Creates `ArrayReader` and `ArrayIterator` concepts: parallels to `RecordBatchReader` and `RecordBatchIterator` that yields `ArrayRef` instead of `RecordBatch`.
- Defines `ArrayStreamReader` to describe an FFI stream of arrays, not of record batches
> [!IMPORTANT]
> Naming is hard here because the existing `ArrowArrayStreamReader` is a bit ambiguously named. It's the core struct for reading from the `ArrowArrayStream` FFI container. But it reads _`RecordBatch`es_, not arrays.
>
> So what do we name the struct that yields arrays? One struct can't implement both `ArrayReader` and `RecordBatchReader` because each trait requires implementation of `Iterator` with a different yield type.
>
> I'm not sure what the best path forward is here.
## Todo:
- [ ] We don't yet support _exporting_ an `ArrayReader` through FFI. I wanted to wait and get feedback on naming and general approach before implementing that.
# Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
If this PR claims a performance improvement, please include evidence such as benchmark results.
-->
# Are there any user-facing changes?
Yes, new APIs.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please call them out.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#issuecomment-5941920272
I did the rename but we still don't support _exporting_ yet.
> * [ ] We don't yet support _exporting_ an `ArrayReader` through FFI.
I'll try to find some time to implement that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4145158389
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any** data type. This
+/// supports reading a stream of generic arrays that may not represent record batches.
Review Comment:
I don't fully undersrtand the distinction -- can't ArrowArrayStreamReader also read any type of array (as RecordBatch can have any type of array 🤔 )
##########
arrow-array/src/array_reader.rs:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Traits for reading a stream of [`ArrayRef`] that share a single [`FieldRef`].
+
+use arrow_schema::{ArrowError, FieldRef};
+
+use crate::array::ArrayRef;
+
+/// Trait for types that can read `ArrayRef`'s.
+///
+/// This is the array-level counterpart to [`RecordBatchReader`], for streams whose
+/// elements are not record batches.
+///
+/// To create from an iterator, see [ArrayIterator].
+///
+/// [`RecordBatchReader`]: crate::RecordBatchReader
+pub trait ArrayReader: Iterator<Item = Result<ArrayRef, ArrowError>> {
+ /// Returns the field of this `ArrayReader`.
+ ///
+ /// Implementation of this trait should guarantee that all `ArrayRef`'s returned by this
+ /// reader should have the same field as returned from this method.
+ fn field(&self) -> FieldRef;
Review Comment:
If possible, I recommend returning `&FieldRef` so the caller can decide if they need to increment the ref count. 🤷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4148226475
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any** data type. This
+/// supports reading a stream of generic arrays that may not represent record batches.
Review Comment:
This is the entire distinction and is important:
- The _`Array`_ based stream reader lets the item passed through the `ArrowArrayStream` be something _other_ than `Struct`
- The _`RecordBatch`_ based stream reader **requires that the data type is `Struct`**, but then allows the _children_ of the StructArray to be anything. I.e. the top-level `StructArray` in the `ArrowArrayStream` is automatically unpacked to a `RecordBatch`.
Currently arrow-rs only defines the latter, so it **only** supports sharing `Struct`-based arrays via FFI.
There are some use cases where users want to share streams of bare arrays that are not `RecordBatch`es.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4149351244
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any** data type. This
+/// supports reading a stream of generic arrays that may not represent record batches.
Review Comment:
Ah, got it -- that makes sense. I forgot that RecordBatch is often treated as a StructArray outside of arrow-rs 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#issuecomment-6200480556
Thanks @kylebarron -- I think this is great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4148226475
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any** data type. This
+/// supports reading a stream of generic arrays that may not represent record batches.
Review Comment:
This is the entire distinction and is important:
- The _`Array`_ based stream reader lets the item passed through the `ArrowArrayStream` be something _other_ than `Struct`
- The _`RecordBatch`_ based stream reader **requires** that the data type is `Struct`, but then allows the _children_ of the StructArray to be anything.
Currently arrow-rs only defines the latter, so it **only** supports sharing `Struct`-based arrays via FFI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4150027046
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any** data type. This
+/// supports reading a stream of generic arrays that may not represent record batches.
Review Comment:
I think it's mostly due to the single FFI container `ArrowArrayStream` being overloaded to handle FFI for both sequences of arrays ("Chunked Array") and sequences of batches ("Table")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4148226475
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any** data type. This
+/// supports reading a stream of generic arrays that may not represent record batches.
Review Comment:
This is the entire distinction and is important:
- The `ArrowArrayStream` sends _arrays_ not record batches through FFI.
- The _`RecordBatch`_ based stream reader **requires that the data type is `Struct`**, but then allows the _children_ of the StructArray to be anything. I.e. the top-level `StructArray` in the `ArrowArrayStream` is automatically unpacked to a `RecordBatch`. This fails for any stream where the data type is not `Struct`.
- The _`Array`_ based stream reader lets the item passed through the `ArrowArrayStream` be something _other_ than `Struct`
Currently arrow-rs only defines the latter, so it **only** supports sharing `Struct`-based arrays via FFI.
There are some use cases where users want to share streams of bare arrays that are not `RecordBatch`es.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4148226475
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -533,6 +548,78 @@ impl RecordBatchReader for ArrowArrayStreamReader {
}
}
+/// An [`ArrayReader`] which imports arrays from an [`FFI_ArrowArrayStream`].
+///
+/// Unlike [`ArrowArrayStreamReader`], the stream's arrays may be of **any** data type. This
+/// supports reading a stream of generic arrays that may not represent record batches.
Review Comment:
This is the entire distinction and is important:
- The `ArrowArrayStream` sends _arrays_ not record batches through FFI.
- The _`RecordBatch`_ based stream reader **requires that the data type is `Struct`**, but then allows the _children_ of the StructArray to be anything. I.e. the top-level `StructArray` in the `ArrowArrayStream` is automatically unpacked to a `RecordBatch`. This fails for any stream where the data type is not `Struct`.
- The _`Array`_ based stream reader lets the item passed through the `ArrowArrayStream` be something _other_ than `Struct`
Currently arrow-rs only defines the RecordBatch-based stream reader, so it **only** supports sharing `Struct`-based arrays via FFI.
There are some use cases where users want to share streams of bare arrays that are not `RecordBatch`es.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#discussion_r4190371876
##########
arrow-array/src/array_reader.rs:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Traits for reading a stream of [`ArrayRef`] that share a single [`FieldRef`].
+
+use arrow_schema::{ArrowError, FieldRef};
+
+use crate::array::ArrayRef;
+
+/// Trait for types that can read `ArrayRef`'s.
+///
+/// This is the array-level counterpart to [`RecordBatchReader`], for streams whose
+/// elements are not record batches.
+///
+/// To create from an iterator, see [ArrayIterator].
+///
+/// [`RecordBatchReader`]: crate::RecordBatchReader
+pub trait ArrayReader: Iterator<Item = Result<ArrayRef, ArrowError>> {
+ /// Returns the field of this `ArrayReader`.
+ ///
+ /// Implementation of this trait should guarantee that all `ArrayRef`'s returned by this
+ /// reader should have the same field as returned from this method.
Review Comment:
The double "should" could be reduced:
```suggestion
/// reader have the same field as returned from this method.
```
##########
arrow-array/src/array_reader.rs:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Traits for reading a stream of [`ArrayRef`] that share a single [`FieldRef`].
+
+use arrow_schema::{ArrowError, FieldRef};
+
+use crate::array::ArrayRef;
+
+/// Trait for types that can read `ArrayRef`'s.
+///
+/// This is the array-level counterpart to [`RecordBatchReader`], for streams whose
+/// elements are not record batches.
+///
+/// To create from an iterator, see [ArrayIterator].
+///
+/// [`RecordBatchReader`]: crate::RecordBatchReader
+pub trait ArrayReader: Iterator<Item = Result<ArrayRef, ArrowError>> {
Review Comment:
name is unused 👍
https://docs.rs/arrow/latest/arrow/index.html?search=ArrayReader
##########
arrow-array/src/array_reader.rs:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Traits for reading a stream of [`ArrayRef`] that share a single [`FieldRef`].
+
+use arrow_schema::{ArrowError, FieldRef};
+
+use crate::array::ArrayRef;
+
+/// Trait for types that can read `ArrayRef`'s.
+///
+/// This is the array-level counterpart to [`RecordBatchReader`], for streams whose
+/// elements are not record batches.
+///
+/// To create from an iterator, see [ArrayIterator].
+///
+/// [`RecordBatchReader`]: crate::RecordBatchReader
+pub trait ArrayReader: Iterator<Item = Result<ArrayRef, ArrowError>> {
+ /// Returns the field of this `ArrayReader`.
+ ///
+ /// Implementation of this trait should guarantee that all `ArrayRef`'s returned by this
+ /// reader should have the same field as returned from this method.
+ fn field(&self) -> &FieldRef;
+}
+
+impl<R: ArrayReader + ?Sized> ArrayReader for Box<R> {
+ fn field(&self) -> &FieldRef {
+ self.as_ref().field()
+ }
+}
+
+/// Generic implementation of [ArrayReader] that wraps an iterator.
+///
+/// # Example
+///
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_array::{ArrayIterator, ArrayReader, ArrayRef, Int32Array};
+/// # use arrow_schema::{DataType, Field};
+/// #
+/// let a: ArrayRef = Arc::new(Int32Array::from(vec![1, 2]));
+/// let b: ArrayRef = Arc::new(Int32Array::from(vec![3, 4]));
+///
+/// let field = Arc::new(Field::new("values", DataType::Int32, false));
+/// let mut reader = ArrayIterator::new(vec![a.clone(), b.clone()].into_iter().map(Ok), field.clone());
+///
+/// assert_eq!(reader.field(), &field);
+/// assert_eq!(&reader.next().unwrap().unwrap(), &a);
+/// # assert_eq!(&reader.next().unwrap().unwrap(), &b);
+/// # assert!(reader.next().is_none());
+/// ```
+pub struct ArrayIterator<I>
Review Comment:
similarly, available: https://docs.rs/arrow/latest/arrow/index.html?search=ArrayIterator
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -21,14 +21,15 @@
//! One interface maps C ABI to native Rust types, i.e. convert c-pointers, c_char, to native rust.
//! This is handled by [FFI_ArrowArrayStream].
//!
-//! The second interface is used to import `FFI_ArrowArrayStream` as Rust implementation `RecordBatch` reader.
-//! This is handled by `ArrowArrayStreamReader`.
+//! The second interface is used to import `FFI_ArrowArrayStream` as a Rust reader. A stream
+//! whose top-level type is `Struct` can be read as `RecordBatch` by `RecordBatchStreamReader`;
+//! a stream of any type can be read as `ArrayRef` by `ArrayStreamReader`.
Review Comment:
can we please make `ArrayStreamReader` a doc reference? [`ArrayStreamReader`] ?
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -404,11 +407,15 @@ fn get_error_code(err: &ArrowError) -> i32 {
/// Its main responsibility is to expose `RecordBatchReader` functionality
/// that requires [FFI_ArrowArrayStream].
#[derive(Debug)]
-pub struct ArrowArrayStreamReader {
+pub struct RecordBatchStreamReader {
Review Comment:
👍
##########
arrow-array/src/ffi_stream.rs:
##########
@@ -404,11 +407,15 @@ fn get_error_code(err: &ArrowError) -> i32 {
/// Its main responsibility is to expose `RecordBatchReader` functionality
/// that requires [FFI_ArrowArrayStream].
#[derive(Debug)]
-pub struct ArrowArrayStreamReader {
+pub struct RecordBatchStreamReader {
stream: FFI_ArrowArrayStream,
schema: SchemaRef,
}
+/// Renamed to [`RecordBatchStreamReader`], to distinguish it from [`ArrayStreamReader`].
+#[deprecated(since = "60.1.0", note = "Use `RecordBatchStreamReader` instead")]
Review Comment:
nice
##########
arrow-array/src/array_reader.rs:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Traits for reading a stream of [`ArrayRef`] that share a single [`FieldRef`].
+
+use arrow_schema::{ArrowError, FieldRef};
+
+use crate::array::ArrayRef;
+
+/// Trait for types that can read `ArrayRef`'s.
+///
+/// This is the array-level counterpart to [`RecordBatchReader`], for streams whose
+/// elements are not record batches.
+///
+/// To create from an iterator, see [ArrayIterator].
+///
+/// [`RecordBatchReader`]: crate::RecordBatchReader
+pub trait ArrayReader: Iterator<Item = Result<ArrayRef, ArrowError>> {
+ /// Returns the field of this `ArrayReader`.
+ ///
+ /// Implementation of this trait should guarantee that all `ArrayRef`'s returned by this
+ /// reader should have the same field as returned from this method.
+ fn field(&self) -> &FieldRef;
+}
+
+impl<R: ArrayReader + ?Sized> ArrayReader for Box<R> {
+ fn field(&self) -> &FieldRef {
+ self.as_ref().field()
+ }
+}
+
+/// Generic implementation of [ArrayReader] that wraps an iterator.
+///
+/// # Example
+///
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_array::{ArrayIterator, ArrayReader, ArrayRef, Int32Array};
+/// # use arrow_schema::{DataType, Field};
+/// #
+/// let a: ArrayRef = Arc::new(Int32Array::from(vec![1, 2]));
+/// let b: ArrayRef = Arc::new(Int32Array::from(vec![3, 4]));
+///
+/// let field = Arc::new(Field::new("values", DataType::Int32, false));
+/// let mut reader = ArrayIterator::new(vec![a.clone(), b.clone()].into_iter().map(Ok), field.clone());
Review Comment:
Possibly easier to understand:
```suggestion
/// let iter = vec![a.clone(), b.clone()].into_iter().map(Ok);
/// let mut reader = ArrayIterator::new(iter, field.clone());
```
##########
arrow-array/src/array_reader.rs:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Traits for reading a stream of [`ArrayRef`] that share a single [`FieldRef`].
+
+use arrow_schema::{ArrowError, FieldRef};
+
+use crate::array::ArrayRef;
+
+/// Trait for types that can read `ArrayRef`'s.
+///
+/// This is the array-level counterpart to [`RecordBatchReader`], for streams whose
+/// elements are not record batches.
+///
+/// To create from an iterator, see [ArrayIterator].
+///
+/// [`RecordBatchReader`]: crate::RecordBatchReader
+pub trait ArrayReader: Iterator<Item = Result<ArrayRef, ArrowError>> {
+ /// Returns the field of this `ArrayReader`.
+ ///
+ /// Implementation of this trait should guarantee that all `ArrayRef`'s returned by this
+ /// reader should have the same field as returned from this method.
+ fn field(&self) -> &FieldRef;
+}
+
+impl<R: ArrayReader + ?Sized> ArrayReader for Box<R> {
+ fn field(&self) -> &FieldRef {
+ self.as_ref().field()
+ }
+}
+
+/// Generic implementation of [ArrayReader] that wraps an iterator.
+///
+/// # Example
+///
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_array::{ArrayIterator, ArrayReader, ArrayRef, Int32Array};
+/// # use arrow_schema::{DataType, Field};
+/// #
+/// let a: ArrayRef = Arc::new(Int32Array::from(vec![1, 2]));
+/// let b: ArrayRef = Arc::new(Int32Array::from(vec![3, 4]));
+///
+/// let field = Arc::new(Field::new("values", DataType::Int32, false));
+/// let mut reader = ArrayIterator::new(vec![a.clone(), b.clone()].into_iter().map(Ok), field.clone());
+///
+/// assert_eq!(reader.field(), &field);
+/// assert_eq!(&reader.next().unwrap().unwrap(), &a);
+/// # assert_eq!(&reader.next().unwrap().unwrap(), &b);
Review Comment:
why hide these?
##########
arrow-array/src/ffi_stream.rs:
##########
Review Comment:
What do you think about cross referencing `ArrayReader` here?
like "See [`ArrayReader`] for reading single arays" or something
And we could add similar "see also" comment to `ArrayReader`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#issuecomment-6200453030
I merged up from main and resolved a clippy issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Introduce `ArrayReader` and `ArrayIterator` concepts [arrow-rs]
Posted by "kylebarron (via GitHub)" <gi...@apache.org>.
kylebarron commented on PR #11161:
URL: https://github.com/apache/arrow-rs/pull/11161#issuecomment-6194877102
I think I've updated for all of your review comments 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org