You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "Rich-T-kid (via GitHub)" <gi...@apache.org> on 2025/09/18 18:04:48 UTC
[PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Rich-T-kid opened a new pull request, #8384:
URL: https://github.com/apache/arrow-rs/pull/8384
# Which issue does this PR close?
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
- Closes #NNN.
# Rationale for this change
[context](https://github.com/apache/arrow-rs/pull/7713)
For whatever reason, I’m running into an authentication issue when trying to push to the branch I created on my Datadog account. I wanted to push some of the refactoring changes that Brancz mentioned on my previous PRs.
I’ve been very busy with school and work, so I don’t think I’ll be able to continue contributing to the project at the same level as I did during the summer. Hopefully, I can still help out with smaller tasks around the codebase, but I don’t have large chunks of time to devote to REE right now.
Sorry about that—I hope I can still make smaller contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#issuecomment-3332442942
@Rich-T-kid I made some comments on the other PR https://github.com/apache/arrow-rs/pull/7713
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed pull request #8384: [Draft] Implemented casting for RunEnd Encoding (pt2)
URL: https://github.com/apache/arrow-rs/pull/8384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#issuecomment-3691203226
Looks like this is superseded by #8589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#issuecomment-3391792170
> @vegarsti you want to open a new PR that combines the changes (of course keeping @Rich-T-kid’s commits in tact but at this point you’ve done with that should be recognized as well :) )?
Yeah, definitely!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on code in PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#discussion_r2384611710
##########
arrow-cast/src/cast/run_array.rs:
##########
@@ -0,0 +1,269 @@
+use crate::cast::*;
+
+/// Attempts to cast a Run-End Encoded array to another type, handling both REE-to-REE
+/// and REE-to-other type conversions with proper validation and error handling.
+///
+/// # Arguments
+/// * `array` - The input Run-End Encoded array to be cast
+/// * `to_type` - The target data type for the casting operation
+/// * `cast_options` - Options controlling the casting behavior (e.g., safe vs unsafe)
+///
+/// # Returns
+/// A `Result` containing the new `ArrayRef` or an `ArrowError` if casting fails
+///
+/// # Behavior
+/// This function handles two main casting scenarios:
+///
+/// ## Case 1: REE-to-REE Casting
+/// When casting to another Run-End Encoded type:
+/// - Casts both the `values` and `run_ends` to their target types
+/// - Validates that run-end casting only allows upcasts (Int16→Int32, Int16→Int64, Int32→Int64)
+/// - Preserves the REE structure while updating both fields
+/// - Returns a new `RunArray` with the appropriate run-end type (Int16, Int32, or Int64)
+///
+/// ## Case 2: REE-to-Other Casting
+/// When casting to a non-REE type:
+/// - Expands the REE array to its logical form by unpacking all values
+/// - Applies the target type casting to the expanded array
+/// - Returns a regular array of the target type (e.g., StringArray, Int64Array)
+///
+/// # Error Handling, error occurs if:
+/// - the input array is not a Run-End Encoded array
+/// - run-end downcasting would cause overflow
+/// - the target run-end type is unsupported
+/// - Propagates errors from underlying casting operations
+///
+/// # Safety Considerations
+/// - Run-end casting uses `safe: false` to prevent silent overflow
+/// - Only upcasts are allowed for run-ends to maintain valid REE structure
+/// - Unpacking preserves null values and array length
+/// - Type validation ensures only supported run-end types (Int16, Int32, Int64)
+///
+/// # Performance Notes
+/// - REE-to-REE casting is efficient as it operates on the compressed structure
+/// - REE-to-other casting requires full unpacking, which may be expensive for large arrays
+/// - Run-end validation adds minimal overhead for safety
+pub(crate) fn run_end_encoded_cast<K: RunEndIndexType>(
+ array: &dyn Array,
+ to_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ match array.data_type() {
+ DataType::RunEndEncoded(_, _) => {
+ let run_array = array
+ .as_any()
+ .downcast_ref::<RunArray<K>>()
+ .ok_or_else(|| ArrowError::CastError("Expected RunArray".to_string()))?;
+
+ let values = run_array.values();
+
+ match to_type {
+ // CASE 1: Stay as RunEndEncoded, cast only the values
+ DataType::RunEndEncoded(target_index_field, target_value_field) => {
+ let cast_values =
+ cast_with_options(values, target_value_field.data_type(), cast_options)?;
+
+ let run_ends_array = PrimitiveArray::<K>::from_iter_values(
+ run_array.run_ends().values().iter().copied(),
+ );
+ let cast_run_ends = cast_with_options(
+ &run_ends_array,
+ target_index_field.data_type(),
+ cast_options,
+ )?;
+ let new_run_array: ArrayRef = match target_index_field.data_type() {
+ DataType::Int16 => {
+ let re = cast_run_ends.as_primitive::<Int16Type>();
+ Arc::new(RunArray::<Int16Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int32 => {
+ let re = cast_run_ends.as_primitive::<Int32Type>();
+ Arc::new(RunArray::<Int32Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int64 => {
+ let re = cast_run_ends.as_primitive::<Int64Type>();
+ Arc::new(RunArray::<Int64Type>::try_new(re, cast_values.as_ref())?)
+ }
+ _ => {
+ return Err(ArrowError::CastError(
+ "Run-end type must be i16, i32, or i64".to_string(),
+ ))
+ }
+ };
+ Ok(Arc::new(new_run_array))
+ }
+
+ // CASE 2: Expand to logical form
+ _ => {
+ let total_len = run_array.len();
+ let indices = Int32Array::from_iter_values(
+ (0..total_len).map(|i| run_array.get_physical_index(i) as i32),
+ );
+
+ let taken = take(values.as_ref(), &indices, None)?;
+
+ if taken.data_type() != to_type {
+ cast_with_options(taken.as_ref(), to_type, cast_options)
+ } else {
+ Ok(taken)
+ }
+ }
+ }
+ }
+
+ _ => Err(ArrowError::CastError(format!(
+ "Cannot cast array of type {:?} to RunEndEncodedArray",
+ array.data_type()
+ ))),
+ }
+}
+
+/// Attempts to cast an array to a RunEndEncoded array with the specified index type K
+/// and value type. This function performs run-end encoding on the input array.
+///
+/// # Arguments
+/// * `array` - The input array to be run-end encoded
+/// * `value_type` - The target data type for the values in the RunEndEncoded array
+/// * `cast_options` - Options controlling the casting behavior
+///
+/// # Returns
+/// A `Result` containing the new `RunArray` or an `ArrowError` if casting fails
+///
+/// # Process
+/// 1. Cast the input array to the target value type if needed
+/// 2. Iterate through the array to identify runs of consecutive equal values
+/// 3. Build run_ends array indicating where each run terminates
+/// 4. Build values array containing the unique values for each run
+/// 5. Construct and return the RunArray
+pub(crate) fn cast_to_run_end_encoded<K: RunEndIndexType>(
+ array: &dyn Array,
+ value_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ // Step 1: Cast the input array to the target value type if necessary
+ let cast_array = if array.data_type() == value_type {
+ // No casting needed, use the array as-is
+ make_array(array.to_data())
+ } else {
+ // Cast to the target value type
+ cast_with_options(array, value_type, cast_options)?
+ };
Review Comment:
```suggestion
let cast_array = if array.data_type() == value_type {
array
} else {
&cast_with_options(array, value_type, cast_options)?
};
```
##########
arrow-cast/src/cast/run_array.rs:
##########
@@ -0,0 +1,269 @@
+use crate::cast::*;
+
+/// Attempts to cast a Run-End Encoded array to another type, handling both REE-to-REE
+/// and REE-to-other type conversions with proper validation and error handling.
+///
+/// # Arguments
+/// * `array` - The input Run-End Encoded array to be cast
+/// * `to_type` - The target data type for the casting operation
+/// * `cast_options` - Options controlling the casting behavior (e.g., safe vs unsafe)
+///
+/// # Returns
+/// A `Result` containing the new `ArrayRef` or an `ArrowError` if casting fails
+///
+/// # Behavior
+/// This function handles two main casting scenarios:
+///
+/// ## Case 1: REE-to-REE Casting
+/// When casting to another Run-End Encoded type:
+/// - Casts both the `values` and `run_ends` to their target types
+/// - Validates that run-end casting only allows upcasts (Int16→Int32, Int16→Int64, Int32→Int64)
+/// - Preserves the REE structure while updating both fields
+/// - Returns a new `RunArray` with the appropriate run-end type (Int16, Int32, or Int64)
+///
+/// ## Case 2: REE-to-Other Casting
+/// When casting to a non-REE type:
+/// - Expands the REE array to its logical form by unpacking all values
+/// - Applies the target type casting to the expanded array
+/// - Returns a regular array of the target type (e.g., StringArray, Int64Array)
+///
+/// # Error Handling, error occurs if:
+/// - the input array is not a Run-End Encoded array
+/// - run-end downcasting would cause overflow
+/// - the target run-end type is unsupported
+/// - Propagates errors from underlying casting operations
+///
+/// # Safety Considerations
+/// - Run-end casting uses `safe: false` to prevent silent overflow
+/// - Only upcasts are allowed for run-ends to maintain valid REE structure
+/// - Unpacking preserves null values and array length
+/// - Type validation ensures only supported run-end types (Int16, Int32, Int64)
+///
+/// # Performance Notes
+/// - REE-to-REE casting is efficient as it operates on the compressed structure
+/// - REE-to-other casting requires full unpacking, which may be expensive for large arrays
+/// - Run-end validation adds minimal overhead for safety
+pub(crate) fn run_end_encoded_cast<K: RunEndIndexType>(
+ array: &dyn Array,
+ to_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ match array.data_type() {
+ DataType::RunEndEncoded(_, _) => {
+ let run_array = array
+ .as_any()
+ .downcast_ref::<RunArray<K>>()
+ .ok_or_else(|| ArrowError::CastError("Expected RunArray".to_string()))?;
+
+ let values = run_array.values();
+
+ match to_type {
+ // CASE 1: Stay as RunEndEncoded, cast only the values
+ DataType::RunEndEncoded(target_index_field, target_value_field) => {
+ let cast_values =
+ cast_with_options(values, target_value_field.data_type(), cast_options)?;
+
+ let run_ends_array = PrimitiveArray::<K>::from_iter_values(
+ run_array.run_ends().values().iter().copied(),
+ );
+ let cast_run_ends = cast_with_options(
+ &run_ends_array,
+ target_index_field.data_type(),
+ cast_options,
+ )?;
+ let new_run_array: ArrayRef = match target_index_field.data_type() {
+ DataType::Int16 => {
+ let re = cast_run_ends.as_primitive::<Int16Type>();
+ Arc::new(RunArray::<Int16Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int32 => {
+ let re = cast_run_ends.as_primitive::<Int32Type>();
+ Arc::new(RunArray::<Int32Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int64 => {
+ let re = cast_run_ends.as_primitive::<Int64Type>();
+ Arc::new(RunArray::<Int64Type>::try_new(re, cast_values.as_ref())?)
+ }
+ _ => {
+ return Err(ArrowError::CastError(
+ "Run-end type must be i16, i32, or i64".to_string(),
+ ))
+ }
+ };
+ Ok(Arc::new(new_run_array))
+ }
+
+ // CASE 2: Expand to logical form
+ _ => {
+ let total_len = run_array.len();
+ let indices = Int32Array::from_iter_values(
+ (0..total_len).map(|i| run_array.get_physical_index(i) as i32),
+ );
+
+ let taken = take(values.as_ref(), &indices, None)?;
+
+ if taken.data_type() != to_type {
+ cast_with_options(taken.as_ref(), to_type, cast_options)
+ } else {
+ Ok(taken)
+ }
+ }
+ }
+ }
+
+ _ => Err(ArrowError::CastError(format!(
+ "Cannot cast array of type {:?} to RunEndEncodedArray",
+ array.data_type()
+ ))),
+ }
+}
+
+/// Attempts to cast an array to a RunEndEncoded array with the specified index type K
+/// and value type. This function performs run-end encoding on the input array.
+///
+/// # Arguments
+/// * `array` - The input array to be run-end encoded
+/// * `value_type` - The target data type for the values in the RunEndEncoded array
+/// * `cast_options` - Options controlling the casting behavior
+///
+/// # Returns
+/// A `Result` containing the new `RunArray` or an `ArrowError` if casting fails
+///
+/// # Process
+/// 1. Cast the input array to the target value type if needed
+/// 2. Iterate through the array to identify runs of consecutive equal values
+/// 3. Build run_ends array indicating where each run terminates
+/// 4. Build values array containing the unique values for each run
+/// 5. Construct and return the RunArray
+pub(crate) fn cast_to_run_end_encoded<K: RunEndIndexType>(
+ array: &dyn Array,
+ value_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ // Step 1: Cast the input array to the target value type if necessary
+ let cast_array = if array.data_type() == value_type {
+ // No casting needed, use the array as-is
+ make_array(array.to_data())
+ } else {
+ // Cast to the target value type
+ cast_with_options(array, value_type, cast_options)?
+ };
+
+ // Step 2: Run-end encode the cast array
+ // We'll use a builder to construct the RunArray efficiently
+ let mut run_ends_builder = PrimitiveBuilder::<K>::new();
+
+ if cast_array.len() == 0 {
+ // Handle empty array case
+ let empty_run_ends = run_ends_builder.finish();
+ let empty_values = make_array(ArrayData::new_empty(value_type));
+ return Ok(Arc::new(RunArray::<K>::try_new(
+ &empty_run_ends,
+ empty_values.as_ref(),
+ )?));
+ }
+
+ // Create a temporary builder to construct the run array
+ // We'll iterate through and build runs by comparing adjacent elements
+ let mut run_ends_vec = Vec::new();
+ let mut values_indices = Vec::new();
+
+ // Add the first element as the start of the first run
+ values_indices.push(0);
+ // Step 3: Identify runs of consecutive equal values
+ for i in 1..cast_array.len() {
+ // We can afford to perform the simple comparison here as we already validated the type in [can_cast_run_end_encoded]
+ let values_equal = match (cast_array.is_null(i), cast_array.is_null(i - 1)) {
+ (true, true) => true, // Both null
+ (false, false) => {
+ // Both non-null - use slice comparison as a basic approach
+ // This is a simplified implementation
+ cast_array.slice(i, 1).to_data() == cast_array.slice(i - 1, 1).to_data()
+ }
+ _ => false, // One null, one not null
+ };
+
+ if !values_equal {
+ // End current run, start new run
+ run_ends_vec.push(i);
+ values_indices.push(i);
+ }
+ }
+
+ // Add the final run end
+ run_ends_vec.push(cast_array.len());
+
+ // Step 4: Build the run_ends array
+ for run_end in run_ends_vec {
+ run_ends_builder.append_value(
+ K::Native::from_usize(run_end)
+ .ok_or_else(|| ArrowError::CastError("Run end index out of range".to_string()))?,
+ );
+ }
+ let run_ends_array = run_ends_builder.finish();
+
+ // Step 5: Build the values array by taking elements at the run start positions
+ let indices = PrimitiveArray::<UInt32Type>::from_iter_values(
+ values_indices.iter().map(|&idx| idx as u32),
+ );
+ let values_array = take(&cast_array, &indices, None)?;
Review Comment:
```suggestion
let values_array = take(cast_array, &indices, None)?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on code in PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#discussion_r2384605923
##########
arrow-cast/src/cast/run_array.rs:
##########
@@ -0,0 +1,269 @@
+use crate::cast::*;
+
+/// Attempts to cast a Run-End Encoded array to another type, handling both REE-to-REE
+/// and REE-to-other type conversions with proper validation and error handling.
+///
+/// # Arguments
+/// * `array` - The input Run-End Encoded array to be cast
+/// * `to_type` - The target data type for the casting operation
+/// * `cast_options` - Options controlling the casting behavior (e.g., safe vs unsafe)
+///
+/// # Returns
+/// A `Result` containing the new `ArrayRef` or an `ArrowError` if casting fails
+///
+/// # Behavior
+/// This function handles two main casting scenarios:
+///
+/// ## Case 1: REE-to-REE Casting
+/// When casting to another Run-End Encoded type:
+/// - Casts both the `values` and `run_ends` to their target types
+/// - Validates that run-end casting only allows upcasts (Int16→Int32, Int16→Int64, Int32→Int64)
+/// - Preserves the REE structure while updating both fields
+/// - Returns a new `RunArray` with the appropriate run-end type (Int16, Int32, or Int64)
+///
+/// ## Case 2: REE-to-Other Casting
+/// When casting to a non-REE type:
+/// - Expands the REE array to its logical form by unpacking all values
+/// - Applies the target type casting to the expanded array
+/// - Returns a regular array of the target type (e.g., StringArray, Int64Array)
+///
+/// # Error Handling, error occurs if:
+/// - the input array is not a Run-End Encoded array
+/// - run-end downcasting would cause overflow
+/// - the target run-end type is unsupported
+/// - Propagates errors from underlying casting operations
+///
+/// # Safety Considerations
+/// - Run-end casting uses `safe: false` to prevent silent overflow
+/// - Only upcasts are allowed for run-ends to maintain valid REE structure
+/// - Unpacking preserves null values and array length
+/// - Type validation ensures only supported run-end types (Int16, Int32, Int64)
+///
+/// # Performance Notes
+/// - REE-to-REE casting is efficient as it operates on the compressed structure
+/// - REE-to-other casting requires full unpacking, which may be expensive for large arrays
+/// - Run-end validation adds minimal overhead for safety
+pub(crate) fn run_end_encoded_cast<K: RunEndIndexType>(
+ array: &dyn Array,
+ to_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ match array.data_type() {
+ DataType::RunEndEncoded(_, _) => {
+ let run_array = array
+ .as_any()
+ .downcast_ref::<RunArray<K>>()
+ .ok_or_else(|| ArrowError::CastError("Expected RunArray".to_string()))?;
+
+ let values = run_array.values();
+
+ match to_type {
+ // CASE 1: Stay as RunEndEncoded, cast only the values
+ DataType::RunEndEncoded(target_index_field, target_value_field) => {
+ let cast_values =
+ cast_with_options(values, target_value_field.data_type(), cast_options)?;
+
+ let run_ends_array = PrimitiveArray::<K>::from_iter_values(
+ run_array.run_ends().values().iter().copied(),
+ );
+ let cast_run_ends = cast_with_options(
+ &run_ends_array,
+ target_index_field.data_type(),
+ cast_options,
+ )?;
+ let new_run_array: ArrayRef = match target_index_field.data_type() {
+ DataType::Int16 => {
+ let re = cast_run_ends.as_primitive::<Int16Type>();
+ Arc::new(RunArray::<Int16Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int32 => {
+ let re = cast_run_ends.as_primitive::<Int32Type>();
+ Arc::new(RunArray::<Int32Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int64 => {
+ let re = cast_run_ends.as_primitive::<Int64Type>();
+ Arc::new(RunArray::<Int64Type>::try_new(re, cast_values.as_ref())?)
+ }
+ _ => {
+ return Err(ArrowError::CastError(
+ "Run-end type must be i16, i32, or i64".to_string(),
+ ))
+ }
+ };
+ Ok(Arc::new(new_run_array))
+ }
+
+ // CASE 2: Expand to logical form
+ _ => {
+ let total_len = run_array.len();
+ let indices = Int32Array::from_iter_values(
+ (0..total_len).map(|i| run_array.get_physical_index(i) as i32),
+ );
+
+ let taken = take(values.as_ref(), &indices, None)?;
+
+ if taken.data_type() != to_type {
+ cast_with_options(taken.as_ref(), to_type, cast_options)
+ } else {
+ Ok(taken)
+ }
+ }
+ }
+ }
+
+ _ => Err(ArrowError::CastError(format!(
+ "Cannot cast array of type {:?} to RunEndEncodedArray",
+ array.data_type()
+ ))),
+ }
+}
+
+/// Attempts to cast an array to a RunEndEncoded array with the specified index type K
+/// and value type. This function performs run-end encoding on the input array.
+///
+/// # Arguments
+/// * `array` - The input array to be run-end encoded
+/// * `value_type` - The target data type for the values in the RunEndEncoded array
+/// * `cast_options` - Options controlling the casting behavior
+///
+/// # Returns
+/// A `Result` containing the new `RunArray` or an `ArrowError` if casting fails
+///
+/// # Process
+/// 1. Cast the input array to the target value type if needed
+/// 2. Iterate through the array to identify runs of consecutive equal values
+/// 3. Build run_ends array indicating where each run terminates
+/// 4. Build values array containing the unique values for each run
+/// 5. Construct and return the RunArray
+pub(crate) fn cast_to_run_end_encoded<K: RunEndIndexType>(
+ array: &dyn Array,
+ value_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ // Step 1: Cast the input array to the target value type if necessary
+ let cast_array = if array.data_type() == value_type {
+ // No casting needed, use the array as-is
+ make_array(array.to_data())
+ } else {
+ // Cast to the target value type
+ cast_with_options(array, value_type, cast_options)?
+ };
+
+ // Step 2: Run-end encode the cast array
+ // We'll use a builder to construct the RunArray efficiently
+ let mut run_ends_builder = PrimitiveBuilder::<K>::new();
+
+ if cast_array.len() == 0 {
+ // Handle empty array case
+ let empty_run_ends = run_ends_builder.finish();
+ let empty_values = make_array(ArrayData::new_empty(value_type));
+ return Ok(Arc::new(RunArray::<K>::try_new(
+ &empty_run_ends,
+ empty_values.as_ref(),
+ )?));
+ }
+
+ // Create a temporary builder to construct the run array
+ // We'll iterate through and build runs by comparing adjacent elements
+ let mut run_ends_vec = Vec::new();
+ let mut values_indices = Vec::new();
+
+ // Add the first element as the start of the first run
+ values_indices.push(0);
+ // Step 3: Identify runs of consecutive equal values
+ for i in 1..cast_array.len() {
+ // We can afford to perform the simple comparison here as we already validated the type in [can_cast_run_end_encoded]
+ let values_equal = match (cast_array.is_null(i), cast_array.is_null(i - 1)) {
+ (true, true) => true, // Both null
+ (false, false) => {
+ // Both non-null - use slice comparison as a basic approach
+ // This is a simplified implementation
+ cast_array.slice(i, 1).to_data() == cast_array.slice(i - 1, 1).to_data()
Review Comment:
```suggestion
// Primitive types
DataType::Boolean => {
cast_array.as_boolean().value(i) == cast_array.as_boolean().value(i - 1)
}
DataType::Int8 => {
cast_array.as_primitive::<Int8Type>().value(i)
== cast_array.as_primitive::<Int8Type>().value(i - 1)
}
DataType::Int16 => {
cast_array.as_primitive::<Int16Type>().value(i)
== cast_array.as_primitive::<Int16Type>().value(i - 1)
}
DataType::Int32 => {
cast_array.as_primitive::<Int32Type>().value(i)
== cast_array.as_primitive::<Int32Type>().value(i - 1)
}
DataType::Int64 => {
cast_array.as_primitive::<Int64Type>().value(i)
== cast_array.as_primitive::<Int64Type>().value(i - 1)
}
DataType::UInt8 => {
cast_array.as_primitive::<UInt8Type>().value(i)
== cast_array.as_primitive::<UInt8Type>().value(i - 1)
}
DataType::UInt16 => {
cast_array.as_primitive::<UInt16Type>().value(i)
== cast_array.as_primitive::<UInt16Type>().value(i - 1)
}
DataType::UInt32 => {
cast_array.as_primitive::<UInt32Type>().value(i)
== cast_array.as_primitive::<UInt32Type>().value(i - 1)
}
DataType::UInt64 => {
cast_array.as_primitive::<UInt64Type>().value(i)
== cast_array.as_primitive::<UInt64Type>().value(i - 1)
}
DataType::Float16 => {
cast_array.as_primitive::<Float16Type>().value(i)
== cast_array.as_primitive::<Float16Type>().value(i - 1)
}
DataType::Float32 => {
cast_array.as_primitive::<Float32Type>().value(i)
== cast_array.as_primitive::<Float32Type>().value(i - 1)
}
DataType::Float64 => {
cast_array.as_primitive::<Float64Type>().value(i)
== cast_array.as_primitive::<Float64Type>().value(i - 1)
}
// String types
DataType::Utf8 => {
cast_array.as_string::<i32>().value(i)
== cast_array.as_string::<i32>().value(i - 1)
}
DataType::LargeUtf8 => {
cast_array.as_string::<i64>().value(i)
== cast_array.as_string::<i64>().value(i - 1)
}
DataType::Utf8View => {
cast_array.as_string_view().value(i) == cast_array.as_string_view().value(i - 1)
}
// Binary types
DataType::Binary => {
cast_array.as_binary::<i32>().value(i)
== cast_array.as_binary::<i32>().value(i - 1)
}
DataType::LargeBinary => {
cast_array.as_binary::<i64>().value(i)
== cast_array.as_binary::<i64>().value(i - 1)
}
DataType::BinaryView => {
cast_array.as_binary_view().value(i) == cast_array.as_binary_view().value(i - 1)
}
DataType::FixedSizeBinary(_) => {
cast_array.as_fixed_size_binary().value(i)
== cast_array.as_fixed_size_binary().value(i - 1)
}
// Temporal types
DataType::Date32 => {
cast_array.as_primitive::<Date32Type>().value(i)
== cast_array.as_primitive::<Date32Type>().value(i - 1)
}
DataType::Date64 => {
cast_array.as_primitive::<Date64Type>().value(i)
== cast_array.as_primitive::<Date64Type>().value(i - 1)
}
DataType::Timestamp(time_unit, _) => match time_unit {
TimeUnit::Second => {
cast_array.as_primitive::<TimestampSecondType>().value(i)
== cast_array
.as_primitive::<TimestampSecondType>()
.value(i - 1)
}
TimeUnit::Millisecond => {
cast_array
.as_primitive::<TimestampMillisecondType>()
.value(i)
== cast_array
.as_primitive::<TimestampMillisecondType>()
.value(i - 1)
}
TimeUnit::Microsecond => {
cast_array
.as_primitive::<TimestampMicrosecondType>()
.value(i)
== cast_array
.as_primitive::<TimestampMicrosecondType>()
.value(i - 1)
}
TimeUnit::Nanosecond => {
cast_array
.as_primitive::<TimestampNanosecondType>()
.value(i)
== cast_array
.as_primitive::<TimestampNanosecondType>()
.value(i - 1)
}
},
DataType::Time32(time_unit) => match time_unit {
TimeUnit::Second => {
cast_array.as_primitive::<Time32SecondType>().value(i)
== cast_array.as_primitive::<Time32SecondType>().value(i - 1)
}
TimeUnit::Millisecond => {
cast_array.as_primitive::<Time32MillisecondType>().value(i)
== cast_array
.as_primitive::<Time32MillisecondType>()
.value(i - 1)
}
TimeUnit::Microsecond | TimeUnit::Nanosecond => {
panic!("Time32 must have a TimeUnit of either seconds or milliseconds")
}
},
DataType::Time64(time_unit) => match time_unit {
TimeUnit::Second | TimeUnit::Millisecond => {
panic!("Time64 must have a TimeUnit of either microseconds or nanoseconds")
}
TimeUnit::Microsecond => {
cast_array.as_primitive::<Time64MicrosecondType>().value(i)
== cast_array
.as_primitive::<Time64MicrosecondType>()
.value(i - 1)
}
TimeUnit::Nanosecond => {
cast_array.as_primitive::<Time64NanosecondType>().value(i)
== cast_array
.as_primitive::<Time64NanosecondType>()
.value(i - 1)
}
},
DataType::Duration(time_unit) => match time_unit {
TimeUnit::Second => {
cast_array.as_primitive::<DurationSecondType>().value(i)
== cast_array.as_primitive::<DurationSecondType>().value(i - 1)
}
TimeUnit::Millisecond => {
cast_array
.as_primitive::<DurationMillisecondType>()
.value(i)
== cast_array
.as_primitive::<DurationMillisecondType>()
.value(i - 1)
}
TimeUnit::Microsecond => {
cast_array
.as_primitive::<DurationMicrosecondType>()
.value(i)
== cast_array
.as_primitive::<DurationMicrosecondType>()
.value(i - 1)
}
TimeUnit::Nanosecond => {
cast_array.as_primitive::<DurationNanosecondType>().value(i)
== cast_array
.as_primitive::<DurationNanosecondType>()
.value(i - 1)
}
},
DataType::Interval(interval_unit) => match interval_unit {
IntervalUnit::YearMonth => {
cast_array.as_primitive::<IntervalYearMonthType>().value(i)
== cast_array
.as_primitive::<IntervalYearMonthType>()
.value(i - 1)
}
IntervalUnit::DayTime => {
cast_array.as_primitive::<IntervalDayTimeType>().value(i)
== cast_array
.as_primitive::<IntervalDayTimeType>()
.value(i - 1)
}
IntervalUnit::MonthDayNano => {
cast_array
.as_primitive::<IntervalMonthDayNanoType>()
.value(i)
== cast_array
.as_primitive::<IntervalMonthDayNanoType>()
.value(i - 1)
}
},
// Decimal types
DataType::Decimal32(_, _) => {
cast_array.as_primitive::<Decimal32Type>().value(i)
== cast_array.as_primitive::<Decimal32Type>().value(i - 1)
}
DataType::Decimal64(_, _) => {
cast_array.as_primitive::<Decimal64Type>().value(i)
== cast_array.as_primitive::<Decimal64Type>().value(i - 1)
}
DataType::Decimal128(_, _) => {
cast_array.as_primitive::<Decimal128Type>().value(i)
== cast_array.as_primitive::<Decimal128Type>().value(i - 1)
}
DataType::Decimal256(_, _) => {
cast_array.as_primitive::<Decimal256Type>().value(i)
== cast_array.as_primitive::<Decimal256Type>().value(i - 1)
}
// TODO: How to handle REE?
DataType::RunEndEncoded(_, _) => todo!(),
DataType::Null
| DataType::List(_)
| DataType::ListView(_)
| DataType::FixedSizeList(_, _)
| DataType::LargeList(_)
| DataType::LargeListView(_)
| DataType::Struct(_)
| DataType::Union(_, _)
| DataType::Dictionary(_, _)
| DataType::Map(_, _) => false,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on code in PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#discussion_r2384605923
##########
arrow-cast/src/cast/run_array.rs:
##########
@@ -0,0 +1,269 @@
+use crate::cast::*;
+
+/// Attempts to cast a Run-End Encoded array to another type, handling both REE-to-REE
+/// and REE-to-other type conversions with proper validation and error handling.
+///
+/// # Arguments
+/// * `array` - The input Run-End Encoded array to be cast
+/// * `to_type` - The target data type for the casting operation
+/// * `cast_options` - Options controlling the casting behavior (e.g., safe vs unsafe)
+///
+/// # Returns
+/// A `Result` containing the new `ArrayRef` or an `ArrowError` if casting fails
+///
+/// # Behavior
+/// This function handles two main casting scenarios:
+///
+/// ## Case 1: REE-to-REE Casting
+/// When casting to another Run-End Encoded type:
+/// - Casts both the `values` and `run_ends` to their target types
+/// - Validates that run-end casting only allows upcasts (Int16→Int32, Int16→Int64, Int32→Int64)
+/// - Preserves the REE structure while updating both fields
+/// - Returns a new `RunArray` with the appropriate run-end type (Int16, Int32, or Int64)
+///
+/// ## Case 2: REE-to-Other Casting
+/// When casting to a non-REE type:
+/// - Expands the REE array to its logical form by unpacking all values
+/// - Applies the target type casting to the expanded array
+/// - Returns a regular array of the target type (e.g., StringArray, Int64Array)
+///
+/// # Error Handling, error occurs if:
+/// - the input array is not a Run-End Encoded array
+/// - run-end downcasting would cause overflow
+/// - the target run-end type is unsupported
+/// - Propagates errors from underlying casting operations
+///
+/// # Safety Considerations
+/// - Run-end casting uses `safe: false` to prevent silent overflow
+/// - Only upcasts are allowed for run-ends to maintain valid REE structure
+/// - Unpacking preserves null values and array length
+/// - Type validation ensures only supported run-end types (Int16, Int32, Int64)
+///
+/// # Performance Notes
+/// - REE-to-REE casting is efficient as it operates on the compressed structure
+/// - REE-to-other casting requires full unpacking, which may be expensive for large arrays
+/// - Run-end validation adds minimal overhead for safety
+pub(crate) fn run_end_encoded_cast<K: RunEndIndexType>(
+ array: &dyn Array,
+ to_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ match array.data_type() {
+ DataType::RunEndEncoded(_, _) => {
+ let run_array = array
+ .as_any()
+ .downcast_ref::<RunArray<K>>()
+ .ok_or_else(|| ArrowError::CastError("Expected RunArray".to_string()))?;
+
+ let values = run_array.values();
+
+ match to_type {
+ // CASE 1: Stay as RunEndEncoded, cast only the values
+ DataType::RunEndEncoded(target_index_field, target_value_field) => {
+ let cast_values =
+ cast_with_options(values, target_value_field.data_type(), cast_options)?;
+
+ let run_ends_array = PrimitiveArray::<K>::from_iter_values(
+ run_array.run_ends().values().iter().copied(),
+ );
+ let cast_run_ends = cast_with_options(
+ &run_ends_array,
+ target_index_field.data_type(),
+ cast_options,
+ )?;
+ let new_run_array: ArrayRef = match target_index_field.data_type() {
+ DataType::Int16 => {
+ let re = cast_run_ends.as_primitive::<Int16Type>();
+ Arc::new(RunArray::<Int16Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int32 => {
+ let re = cast_run_ends.as_primitive::<Int32Type>();
+ Arc::new(RunArray::<Int32Type>::try_new(re, cast_values.as_ref())?)
+ }
+ DataType::Int64 => {
+ let re = cast_run_ends.as_primitive::<Int64Type>();
+ Arc::new(RunArray::<Int64Type>::try_new(re, cast_values.as_ref())?)
+ }
+ _ => {
+ return Err(ArrowError::CastError(
+ "Run-end type must be i16, i32, or i64".to_string(),
+ ))
+ }
+ };
+ Ok(Arc::new(new_run_array))
+ }
+
+ // CASE 2: Expand to logical form
+ _ => {
+ let total_len = run_array.len();
+ let indices = Int32Array::from_iter_values(
+ (0..total_len).map(|i| run_array.get_physical_index(i) as i32),
+ );
+
+ let taken = take(values.as_ref(), &indices, None)?;
+
+ if taken.data_type() != to_type {
+ cast_with_options(taken.as_ref(), to_type, cast_options)
+ } else {
+ Ok(taken)
+ }
+ }
+ }
+ }
+
+ _ => Err(ArrowError::CastError(format!(
+ "Cannot cast array of type {:?} to RunEndEncodedArray",
+ array.data_type()
+ ))),
+ }
+}
+
+/// Attempts to cast an array to a RunEndEncoded array with the specified index type K
+/// and value type. This function performs run-end encoding on the input array.
+///
+/// # Arguments
+/// * `array` - The input array to be run-end encoded
+/// * `value_type` - The target data type for the values in the RunEndEncoded array
+/// * `cast_options` - Options controlling the casting behavior
+///
+/// # Returns
+/// A `Result` containing the new `RunArray` or an `ArrowError` if casting fails
+///
+/// # Process
+/// 1. Cast the input array to the target value type if needed
+/// 2. Iterate through the array to identify runs of consecutive equal values
+/// 3. Build run_ends array indicating where each run terminates
+/// 4. Build values array containing the unique values for each run
+/// 5. Construct and return the RunArray
+pub(crate) fn cast_to_run_end_encoded<K: RunEndIndexType>(
+ array: &dyn Array,
+ value_type: &DataType,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef, ArrowError> {
+ // Step 1: Cast the input array to the target value type if necessary
+ let cast_array = if array.data_type() == value_type {
+ // No casting needed, use the array as-is
+ make_array(array.to_data())
+ } else {
+ // Cast to the target value type
+ cast_with_options(array, value_type, cast_options)?
+ };
+
+ // Step 2: Run-end encode the cast array
+ // We'll use a builder to construct the RunArray efficiently
+ let mut run_ends_builder = PrimitiveBuilder::<K>::new();
+
+ if cast_array.len() == 0 {
+ // Handle empty array case
+ let empty_run_ends = run_ends_builder.finish();
+ let empty_values = make_array(ArrayData::new_empty(value_type));
+ return Ok(Arc::new(RunArray::<K>::try_new(
+ &empty_run_ends,
+ empty_values.as_ref(),
+ )?));
+ }
+
+ // Create a temporary builder to construct the run array
+ // We'll iterate through and build runs by comparing adjacent elements
+ let mut run_ends_vec = Vec::new();
+ let mut values_indices = Vec::new();
+
+ // Add the first element as the start of the first run
+ values_indices.push(0);
+ // Step 3: Identify runs of consecutive equal values
+ for i in 1..cast_array.len() {
+ // We can afford to perform the simple comparison here as we already validated the type in [can_cast_run_end_encoded]
+ let values_equal = match (cast_array.is_null(i), cast_array.is_null(i - 1)) {
+ (true, true) => true, // Both null
+ (false, false) => {
+ // Both non-null - use slice comparison as a basic approach
+ // This is a simplified implementation
+ cast_array.slice(i, 1).to_data() == cast_array.slice(i - 1, 1).to_data()
Review Comment:
Addressing @brancz's comments about not copying https://github.com/apache/arrow-rs/pull/7713#discussion_r2280376316
```suggestion
// Primitive types
DataType::Boolean => {
cast_array.as_boolean().value(i) == cast_array.as_boolean().value(i - 1)
}
DataType::Int8 => {
cast_array.as_primitive::<Int8Type>().value(i)
== cast_array.as_primitive::<Int8Type>().value(i - 1)
}
DataType::Int16 => {
cast_array.as_primitive::<Int16Type>().value(i)
== cast_array.as_primitive::<Int16Type>().value(i - 1)
}
DataType::Int32 => {
cast_array.as_primitive::<Int32Type>().value(i)
== cast_array.as_primitive::<Int32Type>().value(i - 1)
}
DataType::Int64 => {
cast_array.as_primitive::<Int64Type>().value(i)
== cast_array.as_primitive::<Int64Type>().value(i - 1)
}
DataType::UInt8 => {
cast_array.as_primitive::<UInt8Type>().value(i)
== cast_array.as_primitive::<UInt8Type>().value(i - 1)
}
DataType::UInt16 => {
cast_array.as_primitive::<UInt16Type>().value(i)
== cast_array.as_primitive::<UInt16Type>().value(i - 1)
}
DataType::UInt32 => {
cast_array.as_primitive::<UInt32Type>().value(i)
== cast_array.as_primitive::<UInt32Type>().value(i - 1)
}
DataType::UInt64 => {
cast_array.as_primitive::<UInt64Type>().value(i)
== cast_array.as_primitive::<UInt64Type>().value(i - 1)
}
DataType::Float16 => {
cast_array.as_primitive::<Float16Type>().value(i)
== cast_array.as_primitive::<Float16Type>().value(i - 1)
}
DataType::Float32 => {
cast_array.as_primitive::<Float32Type>().value(i)
== cast_array.as_primitive::<Float32Type>().value(i - 1)
}
DataType::Float64 => {
cast_array.as_primitive::<Float64Type>().value(i)
== cast_array.as_primitive::<Float64Type>().value(i - 1)
}
// String types
DataType::Utf8 => {
cast_array.as_string::<i32>().value(i)
== cast_array.as_string::<i32>().value(i - 1)
}
DataType::LargeUtf8 => {
cast_array.as_string::<i64>().value(i)
== cast_array.as_string::<i64>().value(i - 1)
}
DataType::Utf8View => {
cast_array.as_string_view().value(i) == cast_array.as_string_view().value(i - 1)
}
// Binary types
DataType::Binary => {
cast_array.as_binary::<i32>().value(i)
== cast_array.as_binary::<i32>().value(i - 1)
}
DataType::LargeBinary => {
cast_array.as_binary::<i64>().value(i)
== cast_array.as_binary::<i64>().value(i - 1)
}
DataType::BinaryView => {
cast_array.as_binary_view().value(i) == cast_array.as_binary_view().value(i - 1)
}
DataType::FixedSizeBinary(_) => {
cast_array.as_fixed_size_binary().value(i)
== cast_array.as_fixed_size_binary().value(i - 1)
}
// Temporal types
DataType::Date32 => {
cast_array.as_primitive::<Date32Type>().value(i)
== cast_array.as_primitive::<Date32Type>().value(i - 1)
}
DataType::Date64 => {
cast_array.as_primitive::<Date64Type>().value(i)
== cast_array.as_primitive::<Date64Type>().value(i - 1)
}
DataType::Timestamp(time_unit, _) => match time_unit {
TimeUnit::Second => {
cast_array.as_primitive::<TimestampSecondType>().value(i)
== cast_array
.as_primitive::<TimestampSecondType>()
.value(i - 1)
}
TimeUnit::Millisecond => {
cast_array
.as_primitive::<TimestampMillisecondType>()
.value(i)
== cast_array
.as_primitive::<TimestampMillisecondType>()
.value(i - 1)
}
TimeUnit::Microsecond => {
cast_array
.as_primitive::<TimestampMicrosecondType>()
.value(i)
== cast_array
.as_primitive::<TimestampMicrosecondType>()
.value(i - 1)
}
TimeUnit::Nanosecond => {
cast_array
.as_primitive::<TimestampNanosecondType>()
.value(i)
== cast_array
.as_primitive::<TimestampNanosecondType>()
.value(i - 1)
}
},
DataType::Time32(time_unit) => match time_unit {
TimeUnit::Second => {
cast_array.as_primitive::<Time32SecondType>().value(i)
== cast_array.as_primitive::<Time32SecondType>().value(i - 1)
}
TimeUnit::Millisecond => {
cast_array.as_primitive::<Time32MillisecondType>().value(i)
== cast_array
.as_primitive::<Time32MillisecondType>()
.value(i - 1)
}
TimeUnit::Microsecond | TimeUnit::Nanosecond => {
panic!("Time32 must have a TimeUnit of either seconds or milliseconds")
}
},
DataType::Time64(time_unit) => match time_unit {
TimeUnit::Second | TimeUnit::Millisecond => {
panic!("Time64 must have a TimeUnit of either microseconds or nanoseconds")
}
TimeUnit::Microsecond => {
cast_array.as_primitive::<Time64MicrosecondType>().value(i)
== cast_array
.as_primitive::<Time64MicrosecondType>()
.value(i - 1)
}
TimeUnit::Nanosecond => {
cast_array.as_primitive::<Time64NanosecondType>().value(i)
== cast_array
.as_primitive::<Time64NanosecondType>()
.value(i - 1)
}
},
DataType::Duration(time_unit) => match time_unit {
TimeUnit::Second => {
cast_array.as_primitive::<DurationSecondType>().value(i)
== cast_array.as_primitive::<DurationSecondType>().value(i - 1)
}
TimeUnit::Millisecond => {
cast_array
.as_primitive::<DurationMillisecondType>()
.value(i)
== cast_array
.as_primitive::<DurationMillisecondType>()
.value(i - 1)
}
TimeUnit::Microsecond => {
cast_array
.as_primitive::<DurationMicrosecondType>()
.value(i)
== cast_array
.as_primitive::<DurationMicrosecondType>()
.value(i - 1)
}
TimeUnit::Nanosecond => {
cast_array.as_primitive::<DurationNanosecondType>().value(i)
== cast_array
.as_primitive::<DurationNanosecondType>()
.value(i - 1)
}
},
DataType::Interval(interval_unit) => match interval_unit {
IntervalUnit::YearMonth => {
cast_array.as_primitive::<IntervalYearMonthType>().value(i)
== cast_array
.as_primitive::<IntervalYearMonthType>()
.value(i - 1)
}
IntervalUnit::DayTime => {
cast_array.as_primitive::<IntervalDayTimeType>().value(i)
== cast_array
.as_primitive::<IntervalDayTimeType>()
.value(i - 1)
}
IntervalUnit::MonthDayNano => {
cast_array
.as_primitive::<IntervalMonthDayNanoType>()
.value(i)
== cast_array
.as_primitive::<IntervalMonthDayNanoType>()
.value(i - 1)
}
},
// Decimal types
DataType::Decimal32(_, _) => {
cast_array.as_primitive::<Decimal32Type>().value(i)
== cast_array.as_primitive::<Decimal32Type>().value(i - 1)
}
DataType::Decimal64(_, _) => {
cast_array.as_primitive::<Decimal64Type>().value(i)
== cast_array.as_primitive::<Decimal64Type>().value(i - 1)
}
DataType::Decimal128(_, _) => {
cast_array.as_primitive::<Decimal128Type>().value(i)
== cast_array.as_primitive::<Decimal128Type>().value(i - 1)
}
DataType::Decimal256(_, _) => {
cast_array.as_primitive::<Decimal256Type>().value(i)
== cast_array.as_primitive::<Decimal256Type>().value(i - 1)
}
// TODO: How to handle REE?
DataType::RunEndEncoded(_, _) => todo!(),
DataType::Null
| DataType::List(_)
| DataType::ListView(_)
| DataType::FixedSizeList(_, _)
| DataType::LargeList(_)
| DataType::LargeListView(_)
| DataType::Struct(_)
| DataType::Union(_, _)
| DataType::Dictionary(_, _)
| DataType::Map(_, _) => false,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#issuecomment-3391128647
@vegarsti you want to open a new PR that combines the changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Draft] Implemented casting for RunEnd Encoding (pt2) [arrow-rs]
Posted by "vegarsti (via GitHub)" <gi...@apache.org>.
vegarsti commented on PR #8384:
URL: https://github.com/apache/arrow-rs/pull/8384#issuecomment-3392934938
Opened https://github.com/apache/arrow-rs/pull/8589 cc @brancz @Rich-T-kid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org