You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/07/31 19:17:08 UTC
[GitHub] [arrow-rs] alamb opened a new pull request, #4607: Add more docs and examples for ListArray and OffsetsBuffer
alamb opened a new pull request, #4607:
URL: https://github.com/apache/arrow-rs/pull/4607
# Which issue does this PR close?
N/A
# Rationale for this change
I personally find `ListArray` confusing both for myself and to explain to others. Recently I made some pictures while reviewing and discussing with @izveigor in DataFusion https://github.com/apache/arrow-datafusion/pull/6936#discussion_r1279718073 and want to add them to the docs so I don't have to keep referring to them
# What changes are included in this PR?
Add more docs and examples for ListArray and OffsetsBuffer
With these changes I think someone has a plausible chance of understanding how to use `ListArray` without having to refer to the Arrow spec
<img width="997" alt="Screenshot 2023-07-31 at 3 15 47 PM" src="https://github.com/apache/arrow-rs/assets/490673/7c33ef29-9c83-47e0-bba2-4bfe3b9923db">
<img width="1246" alt="Screenshot 2023-07-31 at 3 15 50 PM" src="https://github.com/apache/arrow-rs/assets/490673/565d2c16-77e9-48dc-8187-bab9fb425f58">
# Are there any user-facing changes?
Better docs
No API changes
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1280440567
##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -19,7 +19,43 @@ use crate::buffer::ScalarBuffer;
use crate::{ArrowNativeType, MutableBuffer};
use std::ops::Deref;
-/// A non-empty buffer of monotonically increasing, positive integers
+/// A non-empty buffer of monotonically increasing, positive integers.
+///
+/// [`OffsetBuffer`] are used to represent ranges of offsets. An
+/// `OffsetBuffer` of `N+1` items contains `N` such ranges. The start
+/// offset for element `i` is `offsets[i]` and the end offset is
+/// `offsets[i+1]`. Equal offsets represent an empty range.
+///
+/// # Example
+///
+/// This example shows how 5 distinct ranges, are represented using a
+/// 6 entry `OffsetBuffer`. The first entry `(0, 3)` represents the
+/// three offsets `0, 1, 2` entry `(3,3)` represent no offsets
+/// (e.g. an empty list).
+///
+/// ```text
+/// ┌───────┐ ┌───┐
+/// │ (0,3) │ │ 0 │
+/// ├───────┤ ├───┤
+/// │ (3,3) │ │ 3 │
+/// ├───────┤ ├───┤
+/// │ (3,4) │ │ 4 │
+/// ├───────┤ ├───┤
+/// │ (4,5) │ │ 5 │
+/// ├───────┤ ├───┤
+/// │ (5,7) │ │ 5 │
+/// └───────┘ ├───┤
+/// │ 7 │
+/// └───┘
+///
+/// Offsets Buffer
Review Comment:
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279814512
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 0 │ │ D │ │ 4 │
+/// └─────────────┘ └───────┘ │ └───┘ ├───┤ ├───┤ ├───┤
+/// │ 7 │ │ │ 1 │ │ ? │ │ 5 │
+/// │ Validity └───┘ ├───┤ ├───┤
+/// Logical Logical (nulls) Offsets │ │ 1 │ │ F │ │ 6 │
+/// Values Offsets │ └───┘ └───┘
+/// │ Values │ │
+/// (offsets[i], │ ListArray (Array)
+/// offsets[i+1]) └ ─ ─ ─ ─ ─ ─ ┘ │
+/// └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+///
+///
+/// ```
+///
+/// # Example of constructing a [`ListArray`]
+///
+/// Here is code that constructs the [`ListArray`' in the example above
Review Comment:
```suggestion
/// Here is code that constructs the [`ListArray`] in the example above
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jayzhan211 commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1280681073
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
Review Comment:
I think just leaving `[]` might be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279773652
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
Review Comment:
```suggestion
/// │ NULL │ │ (3,4) │ │ 0 │ │ 3 │ │ │ 1 │ │ C │ │ 2 │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 0 │ │ D │ │ 4 │
Review Comment:
```suggestion
/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 1 │ │ D │ │ 4 │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
Review Comment:
```suggestion
/// │ [D] │ │ (4,5) │ │ 1 │ │ 4 │ │ 0│ │ │ ? │ │ 3 │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
Review Comment:
```suggestion
/// │ NULL │ │ (3,4) │ │ 0 │ │ 3 │ │ │ 1 │ │ C │ │ 2 │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 0 │ │ D │ │ 4 │
Review Comment:
```suggestion
/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 1 │ │ D │ │ 4 │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 0 │ │ D │ │ 4 │
+/// └─────────────┘ └───────┘ │ └───┘ ├───┤ ├───┤ ├───┤
+/// │ 7 │ │ │ 1 │ │ ? │ │ 5 │
Review Comment:
```suggestion
/// │ 7 │ │ │ 0 │ │ ? │ │ 5 │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
Review Comment:
```suggestion
/// │ [D] │ │ (4,5) │ │ 1 │ │ 4 │ │ 0│ │ │ ? │ │ 3 │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 0 │ │ D │ │ 4 │
+/// └─────────────┘ └───────┘ │ └───┘ ├───┤ ├───┤ ├───┤
+/// │ 7 │ │ │ 1 │ │ ? │ │ 5 │
Review Comment:
```suggestion
/// │ 7 │ │ │ 0 │ │ ? │ │ 5 │
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] smiklos commented on pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "smiklos (via GitHub)" <gi...@apache.org>.
smiklos commented on PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#issuecomment-1660292595
Would it make sense to also document `FixedSizeListArray` ? It seems to store values differently than `GenericListArray`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279873817
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 3 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 4 │ │ │ 0 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 1 │ │ D │ │ 4 │
+/// └─────────────┘ └───────┘ │ └───┘ ├───┤ ├───┤ ├───┤
+/// │ 7 │ │ │ 0 │ │ ? │ │ 5 │
+/// │ Validity └───┘ ├───┤ ├───┤
+/// Logical Logical (nulls) Offsets │ │ 1 │ │ F │ │ 6 │
+/// Values Offsets │ └───┘ └───┘
+/// │ Values │ │
+/// (offsets[i], │ ListArray (Array)
+/// offsets[i+1]) └ ─ ─ ─ ─ ─ ─ ┘ │
+/// └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+///
+///
+/// ```
+///
+/// # Example of constructing a [`ListArray`]
+///
+/// Here is code that constructs the [`ListArray`] in the example above
+///
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_array::{builder::ListBuilder, builder::StringBuilder, ArrayRef, StringArray, Array};
+/// #
+/// let values_builder = StringBuilder::new();
+/// let mut builder = ListBuilder::new(values_builder);
+///
+/// // [A, B, C]
+/// builder.values().append_value("A");
+/// builder.values().append_value("B");
+/// builder.values().append_value("C");
+/// builder.append(true);
+///
+/// // [ ] (empty list)
+/// builder.append(true);
+///
+/// // Null
+/// builder.values().append_value("?"); // irrelevant
+/// builder.append(false);
+///
+/// // [D]
+/// builder.values().append_value("D");
+/// builder.append(true);
+///
+/// // [NULL, F]
+/// builder.values().append_null();
+/// builder.values().append_value("F");
+/// builder.append(true);
+///
+/// // Build the array
+/// let array = builder.finish();
+///
+/// // Values is a string array
+/// // "A", "B" "C", "?", "D", NULL, "F"
+/// assert_eq!(
+/// array.values().as_ref(),
+/// &StringArray::from(vec![
+/// Some("A"), Some("B"), Some("C"),
+/// Some("?"), Some("D"), None,
+/// Some("F")
+/// ]) as &dyn Array
Review Comment:
```suggestion
/// ])
```
It should infer this automatically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] smiklos commented on pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "smiklos (via GitHub)" <gi...@apache.org>.
smiklos commented on PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#issuecomment-1660377600
> > FixedSizeListArray stores value the same as ListArray but each row has the same size of array.
>
> This isn't entirely accurate, there is no set of offsets which means that all slots have the same width. In particular this includes nulls, which can cause some interesting challenges.
Based on the following array
```
[[1, 2], Null, [3,4]
```
GenericListArray::values will look like this
```
[1, 2, 3]
```
But FixedSizeArray::values would look like this
```
[1, 2, Null, Null, 3, 4]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1281674160
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
Review Comment:
Good idea -- I will make that change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jayzhan211 commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1280674561
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
Review Comment:
This is a little confusing to me.
`(empty)` is just the explanation to `[]`, which representing `[] is empty array`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] smiklos commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "smiklos (via GitHub)" <gi...@apache.org>.
smiklos commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1281486970
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 3 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 4 │ │ │ 0 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 1 │ │ D │ │ 4 │
+/// └─────────────┘ └───────┘ │ └───┘ ├───┤ ├───┤ ├───┤
+/// │ 7 │ │ │ 0 │ │ ? │ │ 5 │
+/// │ Validity └───┘ ├───┤ ├───┤
+/// Logical Logical (nulls) Offsets │ │ 1 │ │ F │ │ 6 │
+/// Values Offsets │ └───┘ └───┘
+/// │ Values │ │
+/// (offsets[i], │ ListArray (Array)
+/// offsets[i+1]) └ ─ ─ ─ ─ ─ ─ ┘ │
+/// └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+///
+///
+/// ```
+///
+/// # Example of constructing a [`ListArray`]
+///
+/// Here is code that constructs the [`ListArray`] in the example above
+///
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_array::{builder::ListBuilder, builder::StringBuilder, ArrayRef, StringArray, Array};
+/// #
+/// let values_builder = StringBuilder::new();
+/// let mut builder = ListBuilder::new(values_builder);
+///
+/// // [A, B, C]
+/// builder.values().append_value("A");
Review Comment:
I was thinking that perhaps the example doc of using the builder pattern should go to `GenericListBuilder` and here we could use the `from` or `try_new_from_array_data` to construct the array directly from values array and nulls.
FixedSizeListArray ist documented like this, so I'm in between adding builder based code there or keep the two separated. We could just link to the builder based approach from Generic/Fixed list array
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279961487
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
Review Comment:
```suggestion
/// a list which maybe empty, may contain NULL and non-null values,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279816490
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
Review Comment:
Thank you -- this is why I made the code example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#issuecomment-1660215167
I'll merge this later in the week if there are no more comments
cc @jayzhan211 @smiklos and @vincev who may also be interested
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jayzhan211 commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1280674561
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
Review Comment:
This is a little confusing to me.
`(empty)` is just the explanation to `[]`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279964161
##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -19,7 +19,43 @@ use crate::buffer::ScalarBuffer;
use crate::{ArrowNativeType, MutableBuffer};
use std::ops::Deref;
-/// A non-empty buffer of monotonically increasing, positive integers
+/// A non-empty buffer of monotonically increasing, positive integers.
+///
+/// [`OffsetBuffer`] are used to represent ranges of offsets. An
+/// `OffsetBuffer` of `N+1` items contains `N` such ranges. The start
+/// offset for element `i` is `offsets[i]` and the end offset is
+/// `offsets[i+1]`. Equal offsets represent an empty range.
+///
+/// # Example
+///
+/// This example shows how 5 distinct ranges, are represented using a
+/// 6 entry `OffsetBuffer`. The first entry `(0, 3)` represents the
+/// three offsets `0, 1, 2` entry `(3,3)` represent no offsets
Review Comment:
```suggestion
/// 6 entry `OffsetBuffer`. The first entry `(0, 3)` represents the
/// three offsets `0, 1, 2`. The entry `(3,3)` represent no offsets
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] izveigor commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "izveigor (via GitHub)" <gi...@apache.org>.
izveigor commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279771170
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 4 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 5 │ │ │ 1 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 0 │ │ D │ │ 4 │
+/// └─────────────┘ └───────┘ │ └───┘ ├───┤ ├───┤ ├───┤
+/// │ 7 │ │ │ 1 │ │ ? │ │ 5 │
+/// │ Validity └───┘ ├───┤ ├───┤
+/// Logical Logical (nulls) Offsets │ │ 1 │ │ F │ │ 6 │
+/// Values Offsets │ └───┘ └───┘
+/// │ Values │ │
+/// (offsets[i], │ ListArray (Array)
+/// offsets[i+1]) └ ─ ─ ─ ─ ─ ─ ┘ │
+/// └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+///
+///
+/// ```
+///
+/// # Example of constructing a [`ListArray`]
+///
+/// Here is code that constructs the [`ListArray`' in the example above
Review Comment:
/// Here is code that constructs the [`ListArray`] in the example above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279965365
##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -19,7 +19,43 @@ use crate::buffer::ScalarBuffer;
use crate::{ArrowNativeType, MutableBuffer};
use std::ops::Deref;
-/// A non-empty buffer of monotonically increasing, positive integers
+/// A non-empty buffer of monotonically increasing, positive integers.
+///
+/// [`OffsetBuffer`] are used to represent ranges of offsets. An
+/// `OffsetBuffer` of `N+1` items contains `N` such ranges. The start
+/// offset for element `i` is `offsets[i]` and the end offset is
+/// `offsets[i+1]`. Equal offsets represent an empty range.
+///
+/// # Example
+///
+/// This example shows how 5 distinct ranges, are represented using a
+/// 6 entry `OffsetBuffer`. The first entry `(0, 3)` represents the
+/// three offsets `0, 1, 2` entry `(3,3)` represent no offsets
+/// (e.g. an empty list).
+///
+/// ```text
+/// ┌───────┐ ┌───┐
+/// │ (0,3) │ │ 0 │
+/// ├───────┤ ├───┤
+/// │ (3,3) │ │ 3 │
+/// ├───────┤ ├───┤
+/// │ (3,4) │ │ 4 │
+/// ├───────┤ ├───┤
+/// │ (4,5) │ │ 5 │
+/// ├───────┤ ├───┤
+/// │ (5,7) │ │ 5 │
+/// └───────┘ ├───┤
+/// │ 7 │
+/// └───┘
+///
+/// Offsets Buffer
Review Comment:
```suggestion
/// ┌───────┐ ┌───┐
/// │ (0,3) │ │ 0 │
/// ├───────┤ ├───┤
/// │ (3,3) │ │ 3 │
/// ├───────┤ ├───┤
/// │ (3,4) │ │ 3 │
/// ├───────┤ ├───┤
/// │ (4,5) │ │ 4 │
/// ├───────┤ ├───┤
/// │ (5,7) │ │ 5 │
/// └───────┘ ├───┤
/// │ 7 │
/// └───┘
///
/// Offsets Buffer
```
##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -19,7 +19,43 @@ use crate::buffer::ScalarBuffer;
use crate::{ArrowNativeType, MutableBuffer};
use std::ops::Deref;
-/// A non-empty buffer of monotonically increasing, positive integers
+/// A non-empty buffer of monotonically increasing, positive integers.
+///
+/// [`OffsetBuffer`] are used to represent ranges of offsets. An
+/// `OffsetBuffer` of `N+1` items contains `N` such ranges. The start
+/// offset for element `i` is `offsets[i]` and the end offset is
+/// `offsets[i+1]`. Equal offsets represent an empty range.
+///
+/// # Example
+///
+/// This example shows how 5 distinct ranges, are represented using a
+/// 6 entry `OffsetBuffer`. The first entry `(0, 3)` represents the
+/// three offsets `0, 1, 2` entry `(3,3)` represent no offsets
+/// (e.g. an empty list).
+///
+/// ```text
+/// ┌───────┐ ┌───┐
+/// │ (0,3) │ │ 0 │
+/// ├───────┤ ├───┤
+/// │ (3,3) │ │ 3 │
+/// ├───────┤ ├───┤
+/// │ (3,4) │ │ 4 │
+/// ├───────┤ ├───┤
+/// │ (4,5) │ │ 5 │
+/// ├───────┤ ├───┤
+/// │ (5,7) │ │ 5 │
+/// └───────┘ ├───┤
+/// │ 7 │
+/// └───┘
+///
+/// Offsets Buffer
Review Comment:
```suggestion
/// ┌───────┐ ┌───┐
/// │ (0,3) │ │ 0 │
/// ├───────┤ ├───┤
/// │ (3,3) │ │ 3 │
/// ├───────┤ ├───┤
/// │ (3,4) │ │ 3 │
/// ├───────┤ ├───┤
/// │ (4,5) │ │ 4 │
/// ├───────┤ ├───┤
/// │ (5,7) │ │ 5 │
/// └───────┘ ├───┤
/// │ 7 │
/// └───┘
///
/// Offsets Buffer
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jayzhan211 commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1280681073
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
Review Comment:
I think just leaving `[]` might be better. Maybe an explanation in the comment!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] smiklos commented on pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "smiklos (via GitHub)" <gi...@apache.org>.
smiklos commented on PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#issuecomment-1661732252
> > Would it make sense to also document FixedSizeListArray ? It seems to store values differently than GenericListArray
>
> Yes it absolutely would make sense. If you have time that would be great. Otherwise I'll try and make a diagram tomorrow
I added [this pr](https://github.com/apache/arrow-rs/pull/4622), largely based on the art in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279962388
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
Review Comment:
To be consistent in style with below:
```suggestion
/// │ [A, B, C] │
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279961487
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
Review Comment:
```suggestion
/// a list which may be empty, may contain NULL and non-null values,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279961872
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
Review Comment:
```suggestion
/// supported Arrow type. Each element of the `ListArray` itself is
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#issuecomment-1660983868
> Would it make sense to also document FixedSizeListArray ? It seems to store values differently than GenericListArray
Yes it absolutely would make sense. If you have time that would be great. Otherwise I'll try and make a diagram tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] viirya commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1279962388
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
Review Comment:
To be consistent in style with below:
```suggestion
/// │ [A, B, C] │
```
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is a
+/// a list which mayb be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
Review Comment:
To be consistent in style with below:
```suggestion
/// │ [A, B, C] │
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jayzhan211 commented on pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#issuecomment-1660364009
> Would it make sense to also document `FixedSizeListArray` ? It seems to store values differently than `GenericListArray`
FixedSizeListArray stores value the same as ListArray but each row has the same size of array.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1281696247
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
+/// ├─────────────┤
+/// │ NULL │
+/// ├─────────────┤
+/// │ [D] │
+/// ├─────────────┤
+/// │ [NULL, F] │
+/// └─────────────┘
+/// ```
+///
+/// The `values` of this `ListArray`s are stored in a child
+/// [`StringArray`] and the offsets are stored in an [`OffsetBuffer`]
+/// as shown in the following diagram. The logical values and offsets
+/// are shown on the left, and the actual `ListArray` encoding on the right
+///
+/// ```text
+/// ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+/// ┌ ─ ─ ─ ─ ─ ─ ┐ │
+/// ┌─────────────┐ ┌───────┐ │ ┌───┐ ┌───┐ ┌───┐ ┌───┐
+/// │ [A,B,C] │ │ (0,3) │ │ 1 │ │ 0 │ │ │ 1 │ │ A │ │ 0 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [] (empty) │ │ (3,3) │ │ 1 │ │ 3 │ │ │ 1 │ │ B │ │ 1 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ NULL │ │ (3,4) │ │ 0 │ │ 3 │ │ │ 1 │ │ C │ │ 2 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [D] │ │ (4,5) │ │ 1 │ │ 4 │ │ │ 0 │ │ ? │ │ 3 │
+/// ├─────────────┤ ├───────┤ │ ├───┤ ├───┤ ├───┤ ├───┤
+/// │ [NULL, F] │ │ (5,7) │ │ 1 │ │ 5 │ │ │ 1 │ │ D │ │ 4 │
+/// └─────────────┘ └───────┘ │ └───┘ ├───┤ ├───┤ ├───┤
+/// │ 7 │ │ │ 0 │ │ ? │ │ 5 │
+/// │ Validity └───┘ ├───┤ ├───┤
+/// Logical Logical (nulls) Offsets │ │ 1 │ │ F │ │ 6 │
+/// Values Offsets │ └───┘ └───┘
+/// │ Values │ │
+/// (offsets[i], │ ListArray (Array)
+/// offsets[i+1]) └ ─ ─ ─ ─ ─ ─ ┘ │
+/// └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
+///
+///
+/// ```
+///
+/// # Example of constructing a [`ListArray`]
+///
+/// Here is code that constructs the [`ListArray`] in the example above
+///
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_array::{builder::ListBuilder, builder::StringBuilder, ArrayRef, StringArray, Array};
+/// #
+/// let values_builder = StringBuilder::new();
+/// let mut builder = ListBuilder::new(values_builder);
+///
+/// // [A, B, C]
+/// builder.values().append_value("A");
Review Comment:
I was torn on this, as keeping the example (that constructs the data in the diagram) I thought added value. However I think your idea of adding a link to the approach would be good.
I implemented this change in 151564d5ab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jayzhan211 commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1280674561
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
Review Comment:
This is a little confusing to me.
`(empty)` is just the explanation to `[]`? Showing that `[] is empty array`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#issuecomment-1660369704
> FixedSizeListArray stores value the same as ListArray but each row has the same size of array.
This isn't entirely accurate, there is no set of offsets which means that all slots have the same width. In particular this includes nulls, which can cause some interesting challenges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on a diff in pull request #4607: Add more docs and examples for ListArray and OffsetsBuffer
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #4607:
URL: https://github.com/apache/arrow-rs/pull/4607#discussion_r1281674160
##########
arrow-array/src/array/list_array.rs:
##########
@@ -54,11 +54,129 @@ impl OffsetSizeTrait for i64 {
const PREFIX: &'static str = "Large";
}
-/// An array of [variable length arrays](https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout)
+/// An array of [variable length lists], similar to JSON arrays
+/// (e.g. `["A", "B", "C"]`).
///
-/// See [`ListArray`] and [`LargeListArray`]`
+/// Lists are represented using `offsets` into a `values` child
+/// array. Offsets are stored in two adjacent entries of an
+/// [`OffsetBuffer`].
///
-/// See [`GenericListBuilder`](crate::builder::GenericListBuilder) for how to construct a [`GenericListArray`]
+/// Arrow defines [`ListArray`] with `i32` offsets and
+/// [`LargeListArray`]` with `i64` offsets.
+///
+/// Use [`GenericListBuilder`](crate::builder::GenericListBuilder) to
+/// construct a [`GenericListArray`].
+///
+/// # Representation
+///
+/// A [`ListArray`] can represent a list of values of any other
+/// supported Arrow type. Each element of the `ListArray` itself is
+/// a list which may be empty, may contain NULL and non-null values,
+/// or may itself be NULL.
+///
+/// For example, this `ListArray` stores lists of strings:
+///
+/// ```text
+/// ┌─────────────┐
+/// │ [A,B,C] │
+/// ├─────────────┤
+/// │ [] (empty) │
Review Comment:
Good idea -- I will make that change.
Edit: Done in [8e6a56d](https://github.com/apache/arrow-rs/pull/4607/commits/8e6a56d1cde1cd5cf5a7a25e16f7e5f241cd51b9)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org