You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2026/10/03 03:44:53 UTC
(arrow-rs) branch gh-readonly-queue/main/pr-11077-7c88505f9e3bb5b5275aa2838e7ef0757962964b created (now 1d1607a761)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-11077-7c88505f9e3bb5b5275aa2838e7ef0757962964b
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
at 1d1607a761 Move binary and byte-view cast helpers into a private submodule (#11077)
This branch includes the following new commits:
new 1d1607a761 Move binary and byte-view cast helpers into a private submodule (#11077)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(arrow-rs) 01/01: Move binary and byte-view cast helpers into a private submodule (#11077)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-11077-7c88505f9e3bb5b5275aa2838e7ef0757962964b
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
commit 1d1607a761e1dff1a104aefcf6ff248f2579818c
Author: Anand Hegde <an...@gmail.com>
AuthorDate: Sat Oct 3 03:44:27 2026 +0000
Move binary and byte-view cast helpers into a private submodule (#11077)
# Which issue does this PR close?
Part of #11032 — the **binary, fixed-size binary and byte-view helpers**
group
only, so it does not close the issue.
Sibling PRs against the same issue: #11075 (temporal) and #11076
(numeric and
boolean). All three are independent and move disjoint functions, but
each adds a
`mod` line and an import to `mod.rs`, so whichever lands later will want
a
trivial rebase.
# Rationale for this change
`arrow-cast/src/cast/mod.rs` is ~14k lines. #5125 moved list, decimal,
dictionary, string, map, run-array and union support into private
submodules,
and #11032 lists the groups still inline. The byte-oriented conversions
are one
of them.
# What changes are included in this PR?
A new private `arrow-cast/src/cast/binary.rs` holding:
- `cast_numeric_to_binary`
- `cast_binary_to_fixed_size_binary`
- `cast_fixed_size_binary_to_binary`
- `cast_fixed_size_binary_to_binary_view`
- `cast_byte_container`
- `cast_view_to_byte`
`cast_with_options` stays the top-level dispatcher in `mod.rs`.
Worth flagging for review:
- **No public API surface is involved here.** Unlike the numeric group
in
#11076, none of these six were `pub`, so `mod.rs` uses a plain
`use crate::cast::binary::*` rather than a `pub use`. All six are called
from
the dispatcher, so all six became `pub(crate)` and nothing stayed
private.
- **`cast_numeric_to_binary` is not adjacent to the rest.** It sits up
with the
numeric helpers in `mod.rs` rather than with the other byte conversions,
but
it belongs to this group, so it is moved from there.
- Three imports left orphaned in `mod.rs` by the move are dropped:
`arrow_data::ByteView`, and `Buffer` from the `arrow_buffer` import
list.
# Are these changes tested?
Covered by the existing `arrow-cast` tests — this is a code move, so no
new
tests are warranted and no existing test was modified.
```
cargo test -p arrow-cast 379 passed + 12 passed, 0 failed (same as base)
cargo clippy -p arrow-cast --all-targets -- -D warnings clean
cargo fmt -p arrow-cast -- --check clean
cargo build -p arrow builds
```
As with the sibling PRs, the faithfulness of the move is checked
mechanically
rather than by eye: extracting the two original line ranges from `HEAD`
and
normalising away the added `pub(crate)` gives a **byte-identical** match
against
the new file.
Run on macOS 26.6.2, aarch64-apple-darwin, toolchain 1.98.1 from
`rust-toolchain.toml`.
# Are there any user-facing changes?
No. No public API change and no behavioural change.
# AI usage disclosure
Written with AI assistance (Claude). The AI performed the mechanical
extraction
and ran the verification above; I reviewed the result and own the
change.
The moved bodies are unedited, and the byte-identity check is the
evidence for
that rather than a reading. The judgement calls — pulling
`cast_numeric_to_binary`
out of the numeric neighbourhood, the uniform `pub(crate)`, and the
three dropped
imports — are listed above because they are the parts a "pure move" diff
would
not otherwise draw attention to.
---
arrow-cast/src/cast/binary.rs | 198 ++++++++++++++++++++++++++++++++++++++++++
arrow-cast/src/cast/mod.rs | 176 +------------------------------------
2 files changed, 201 insertions(+), 173 deletions(-)
diff --git a/arrow-cast/src/cast/binary.rs b/arrow-cast/src/cast/binary.rs
new file mode 100644
index 0000000000..72ce5ae1a9
--- /dev/null
+++ b/arrow-cast/src/cast/binary.rs
@@ -0,0 +1,198 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Cast support for binary, fixed-size binary and byte-view arrays.
+
+use arrow_array::{builder::*, cast::*, types::*, *};
+use arrow_buffer::{Buffer, OffsetBuffer};
+use arrow_data::{ArrayData, ByteView};
+use arrow_schema::{ArrowError, DataType};
+use num_traits::{NumCast, ToPrimitive};
+use std::sync::Arc;
+
+use super::CastOptions;
+
+pub(crate) fn cast_numeric_to_binary<FROM: ArrowPrimitiveType, O: OffsetSizeTrait>(
+ array: &dyn Array,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array.as_primitive::<FROM>();
+ let size = std::mem::size_of::<FROM::Native>();
+ let offsets = OffsetBuffer::from_repeated_length(size, array.len());
+ Ok(Arc::new(GenericBinaryArray::<O>::try_new(
+ offsets,
+ array.values().inner().clone(),
+ array.nulls().cloned(),
+ )?))
+}
+
+/// Helper function to cast from one `BinaryArray` or 'LargeBinaryArray' to 'FixedSizeBinaryArray'.
+pub(crate) fn cast_binary_to_fixed_size_binary<O: OffsetSizeTrait>(
+ array: &dyn Array,
+ byte_width: i32,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array.as_binary::<O>();
+ let mut builder = FixedSizeBinaryBuilder::with_capacity(array.len(), byte_width);
+
+ for i in 0..array.len() {
+ if array.is_null(i) {
+ builder.append_null();
+ } else {
+ match builder.append_value(array.value(i)) {
+ Ok(()) => {}
+ Err(e) => match cast_options.safe {
+ true => builder.append_null(),
+ false => return Err(e),
+ },
+ }
+ }
+ }
+
+ Ok(Arc::new(builder.finish()))
+}
+
+/// Helper function to cast from 'FixedSizeBinaryArray' to one `BinaryArray` or 'LargeBinaryArray'.
+/// If the target one is too large for the source array it will return an Error.
+pub(crate) fn cast_fixed_size_binary_to_binary<O: OffsetSizeTrait>(
+ array: &dyn Array,
+ byte_width: i32,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array
+ .as_any()
+ .downcast_ref::<FixedSizeBinaryArray>()
+ .unwrap();
+
+ let offsets: i128 = byte_width as i128 * array.len() as i128;
+
+ let is_binary = matches!(GenericBinaryType::<O>::DATA_TYPE, DataType::Binary);
+ if is_binary && offsets > i32::MAX as i128 {
+ return Err(ArrowError::ComputeError(
+ "FixedSizeBinary array too large to cast to Binary array".to_string(),
+ ));
+ } else if !is_binary && offsets > i64::MAX as i128 {
+ return Err(ArrowError::ComputeError(
+ "FixedSizeBinary array too large to cast to LargeBinary array".to_string(),
+ ));
+ }
+
+ let mut builder = GenericBinaryBuilder::<O>::with_capacity(array.len(), array.len());
+
+ for i in 0..array.len() {
+ if array.is_null(i) {
+ builder.append_null();
+ } else {
+ builder.append_value(array.value(i));
+ }
+ }
+
+ Ok(Arc::new(builder.finish()))
+}
+
+pub(crate) fn cast_fixed_size_binary_to_binary_view(
+ array: &dyn Array,
+ _byte_width: i32,
+) -> Result<ArrayRef, ArrowError> {
+ let array = array
+ .as_any()
+ .downcast_ref::<FixedSizeBinaryArray>()
+ .unwrap();
+
+ let mut builder = BinaryViewBuilder::with_capacity(array.len());
+ for i in 0..array.len() {
+ if array.is_null(i) {
+ builder.append_null();
+ } else {
+ builder.append_value(array.value(i));
+ }
+ }
+
+ Ok(Arc::new(builder.finish()))
+}
+
+/// Helper function to cast from one `ByteArrayType` to another and vice versa.
+/// If the target one (e.g., `LargeUtf8`) is too large for the source array it will return an Error.
+pub(crate) fn cast_byte_container<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError>
+where
+ FROM: ByteArrayType,
+ TO: ByteArrayType<Native = FROM::Native>,
+ FROM::Offset: OffsetSizeTrait + ToPrimitive,
+ TO::Offset: OffsetSizeTrait + NumCast,
+{
+ let data = array.to_data();
+ assert_eq!(data.data_type(), &FROM::DATA_TYPE);
+ let str_values_buf = data.buffers()[1].clone();
+ let offsets = data.buffers()[0].typed_data::<FROM::Offset>();
+
+ let mut cast_offsets = Vec::<TO::Offset>::with_capacity(offsets.len());
+ offsets
+ .iter()
+ .try_for_each::<_, Result<_, ArrowError>>(|offset| {
+ let offset =
+ <<TO as ByteArrayType>::Offset as NumCast>::from(*offset).ok_or_else(|| {
+ ArrowError::ComputeError(format!(
+ "{}{} array too large to cast to {}{} array",
+ FROM::Offset::PREFIX,
+ FROM::PREFIX,
+ TO::Offset::PREFIX,
+ TO::PREFIX
+ ))
+ })?;
+ cast_offsets.push(offset);
+ Ok(())
+ })?;
+
+ let offset_buffer = Buffer::from_vec(cast_offsets);
+
+ let dtype = TO::DATA_TYPE;
+
+ let builder = ArrayData::builder(dtype)
+ .offset(array.offset())
+ .len(array.len())
+ .add_buffer(offset_buffer)
+ .add_buffer(str_values_buf)
+ .nulls(data.nulls().cloned());
+
+ let array_data = unsafe { builder.build_unchecked() };
+
+ Ok(Arc::new(GenericByteArray::<TO>::from(array_data)))
+}
+
+/// Helper function to cast from one `ByteViewType` array to `ByteArrayType` array.
+pub(crate) fn cast_view_to_byte<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError>
+where
+ FROM: ByteViewType,
+ TO: ByteArrayType,
+ FROM::Native: AsRef<TO::Native>,
+{
+ let data = array.to_data();
+ let view_array = GenericByteViewArray::<FROM>::from(data);
+
+ let len = view_array.len();
+ let bytes = view_array
+ .views()
+ .iter()
+ .map(|v| ByteView::from(*v).length as usize)
+ .sum::<usize>();
+
+ let mut byte_array_builder = GenericByteBuilder::<TO>::with_capacity(len, bytes);
+
+ for val in &view_array {
+ byte_array_builder.append_option(val);
+ }
+
+ Ok(Arc::new(byte_array_builder.finish()))
+}
diff --git a/arrow-cast/src/cast/mod.rs b/arrow-cast/src/cast/mod.rs
index 4bff879292..0887345d05 100644
--- a/arrow-cast/src/cast/mod.rs
+++ b/arrow-cast/src/cast/mod.rs
@@ -37,6 +37,7 @@
//! assert_eq!(7.0, c.value(2));
//! ```
+mod binary;
mod decimal;
mod dictionary;
mod list;
@@ -46,6 +47,7 @@ mod string;
mod structs;
mod union;
+use crate::cast::binary::*;
use crate::cast::decimal::*;
use crate::cast::dictionary::*;
use crate::cast::list::*;
@@ -56,7 +58,6 @@ use crate::cast::structs::*;
pub use crate::cast::union::*;
use arrow_buffer::IntervalMonthDayNano;
-use arrow_data::ByteView;
use chrono::{
FixedOffset, LocalResult, NaiveDateTime, NaiveTime, Offset, TimeDelta, TimeZone, Utc,
};
@@ -69,7 +70,7 @@ use crate::parse::{
string_to_datetime,
};
use arrow_array::{builder::*, cast::*, temporal_conversions::*, timezone::Tz, types::*, *};
-use arrow_buffer::{ArrowNativeType, Buffer, OffsetBuffer, i256};
+use arrow_buffer::{ArrowNativeType, OffsetBuffer, i256};
use arrow_data::ArrayData;
use arrow_data::transform::MutableArrayData;
use arrow_schema::*;
@@ -2632,19 +2633,6 @@ where
num_traits::cast::cast::<I, O>(value)
}
-fn cast_numeric_to_binary<FROM: ArrowPrimitiveType, O: OffsetSizeTrait>(
- array: &dyn Array,
-) -> Result<ArrayRef, ArrowError> {
- let array = array.as_primitive::<FROM>();
- let size = std::mem::size_of::<FROM::Native>();
- let offsets = OffsetBuffer::from_repeated_length(size, array.len());
- Ok(Arc::new(GenericBinaryArray::<O>::try_new(
- offsets,
- array.values().inner().clone(),
- array.nulls().cloned(),
- )?))
-}
-
/// Returns the offset to use when interpreting `local` as a wall clock reading
/// in `tz`, or `None` if it cannot be resolved.
///
@@ -2801,164 +2789,6 @@ where
}
}
-/// Helper function to cast from one `BinaryArray` or 'LargeBinaryArray' to 'FixedSizeBinaryArray'.
-fn cast_binary_to_fixed_size_binary<O: OffsetSizeTrait>(
- array: &dyn Array,
- byte_width: i32,
- cast_options: &CastOptions,
-) -> Result<ArrayRef, ArrowError> {
- let array = array.as_binary::<O>();
- let mut builder = FixedSizeBinaryBuilder::with_capacity(array.len(), byte_width);
-
- for i in 0..array.len() {
- if array.is_null(i) {
- builder.append_null();
- } else {
- match builder.append_value(array.value(i)) {
- Ok(()) => {}
- Err(e) => match cast_options.safe {
- true => builder.append_null(),
- false => return Err(e),
- },
- }
- }
- }
-
- Ok(Arc::new(builder.finish()))
-}
-
-/// Helper function to cast from 'FixedSizeBinaryArray' to one `BinaryArray` or 'LargeBinaryArray'.
-/// If the target one is too large for the source array it will return an Error.
-fn cast_fixed_size_binary_to_binary<O: OffsetSizeTrait>(
- array: &dyn Array,
- byte_width: i32,
-) -> Result<ArrayRef, ArrowError> {
- let array = array
- .as_any()
- .downcast_ref::<FixedSizeBinaryArray>()
- .unwrap();
-
- let offsets: i128 = byte_width as i128 * array.len() as i128;
-
- let is_binary = matches!(GenericBinaryType::<O>::DATA_TYPE, DataType::Binary);
- if is_binary && offsets > i32::MAX as i128 {
- return Err(ArrowError::ComputeError(
- "FixedSizeBinary array too large to cast to Binary array".to_string(),
- ));
- } else if !is_binary && offsets > i64::MAX as i128 {
- return Err(ArrowError::ComputeError(
- "FixedSizeBinary array too large to cast to LargeBinary array".to_string(),
- ));
- }
-
- let mut builder = GenericBinaryBuilder::<O>::with_capacity(array.len(), array.len());
-
- for i in 0..array.len() {
- if array.is_null(i) {
- builder.append_null();
- } else {
- builder.append_value(array.value(i));
- }
- }
-
- Ok(Arc::new(builder.finish()))
-}
-
-fn cast_fixed_size_binary_to_binary_view(
- array: &dyn Array,
- _byte_width: i32,
-) -> Result<ArrayRef, ArrowError> {
- let array = array
- .as_any()
- .downcast_ref::<FixedSizeBinaryArray>()
- .unwrap();
-
- let mut builder = BinaryViewBuilder::with_capacity(array.len());
- for i in 0..array.len() {
- if array.is_null(i) {
- builder.append_null();
- } else {
- builder.append_value(array.value(i));
- }
- }
-
- Ok(Arc::new(builder.finish()))
-}
-
-/// Helper function to cast from one `ByteArrayType` to another and vice versa.
-/// If the target one (e.g., `LargeUtf8`) is too large for the source array it will return an Error.
-fn cast_byte_container<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError>
-where
- FROM: ByteArrayType,
- TO: ByteArrayType<Native = FROM::Native>,
- FROM::Offset: OffsetSizeTrait + ToPrimitive,
- TO::Offset: OffsetSizeTrait + NumCast,
-{
- let data = array.to_data();
- assert_eq!(data.data_type(), &FROM::DATA_TYPE);
- let str_values_buf = data.buffers()[1].clone();
- let offsets = data.buffers()[0].typed_data::<FROM::Offset>();
-
- let mut cast_offsets = Vec::<TO::Offset>::with_capacity(offsets.len());
- offsets
- .iter()
- .try_for_each::<_, Result<_, ArrowError>>(|offset| {
- let offset =
- <<TO as ByteArrayType>::Offset as NumCast>::from(*offset).ok_or_else(|| {
- ArrowError::ComputeError(format!(
- "{}{} array too large to cast to {}{} array",
- FROM::Offset::PREFIX,
- FROM::PREFIX,
- TO::Offset::PREFIX,
- TO::PREFIX
- ))
- })?;
- cast_offsets.push(offset);
- Ok(())
- })?;
-
- let offset_buffer = Buffer::from_vec(cast_offsets);
-
- let dtype = TO::DATA_TYPE;
-
- let builder = ArrayData::builder(dtype)
- .offset(array.offset())
- .len(array.len())
- .add_buffer(offset_buffer)
- .add_buffer(str_values_buf)
- .nulls(data.nulls().cloned());
-
- let array_data = unsafe { builder.build_unchecked() };
-
- Ok(Arc::new(GenericByteArray::<TO>::from(array_data)))
-}
-
-/// Helper function to cast from one `ByteViewType` array to `ByteArrayType` array.
-fn cast_view_to_byte<FROM, TO>(array: &dyn Array) -> Result<ArrayRef, ArrowError>
-where
- FROM: ByteViewType,
- TO: ByteArrayType,
- FROM::Native: AsRef<TO::Native>,
-{
- let data = array.to_data();
- let view_array = GenericByteViewArray::<FROM>::from(data);
-
- let len = view_array.len();
- let bytes = view_array
- .views()
- .iter()
- .map(|v| ByteView::from(*v).length as usize)
- .sum::<usize>();
-
- let mut byte_array_builder = GenericByteBuilder::<TO>::with_capacity(len, bytes);
-
- for val in &view_array {
- byte_array_builder.append_option(val);
- }
-
- Ok(Arc::new(byte_array_builder.finish()))
-}
-
#[cfg(test)]
mod tests {
use super::*;