You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "asubiotto (via GitHub)" <gi...@apache.org> on 2026/03/12 14:11:37 UTC
[PR] physical-expr-adapter: support casting structs nested inside complex types [datafusion]
asubiotto opened a new pull request, #20907:
URL: https://github.com/apache/datafusion/pull/20907
Previously, DefaultPhysicalExprAdapterRewriter.create_cast_column_expr would only allow struct evolution for top-level structs. However, it is valid to have structs that are nested inside other complex types (e.g. Dictionary<_, Struct>, List<Struct>). These cases would previously return an error although they were valid.
This commit handles Structs that are arbitrarily nested within certain complex types (Lists, ListViews, and Dicts). Other complex types can be supported in the future, but I think this is good enough for now.
## Which issue does this PR close?
- Closes #20906
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Struct evolution is supported, this PR fixes an edge case where struct evolution is not supported when structs are nested inside other complex types.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
Yes, I initially wrote parametrized tests, but I think that's probably overkill.
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
Queries that would previously error, now complete successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix[physical-expr-adapter]: support casting structs nested inside complex types [datafusion]
Posted by "asubiotto (via GitHub)" <gi...@apache.org>.
asubiotto commented on PR #20907:
URL: https://github.com/apache/datafusion/pull/20907#issuecomment-4053750553
Thanks for the review! I will address the comments and have a new version by the middle of next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix[physical-expr-adapter]: support casting structs nested inside complex types [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #20907:
URL: https://github.com/apache/datafusion/pull/20907#discussion_r2929282085
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -1010,4 +1173,18 @@ mod tests {
assert!(b_col.is_null(0));
assert!(b_col.is_null(1));
}
+
Review Comment:
The new functionality adds fairly intricate `ListView` and dictionary rebuild logic, but the new tests exercise only the `List<Struct>` execution path and dictionary compatibility validation.
Do you think It would be worth adding tests for `Dictionary<_, Struct>` and for `ListView<Struct>`?
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -165,18 +170,139 @@ fn cast_struct_column(
/// - Invalid data type combinations are encountered
pub fn cast_column(
source_col: &ArrayRef,
- target_field: &Field,
+ target_type: &DataType,
cast_options: &CastOptions,
) -> Result<ArrayRef> {
- match target_field.data_type() {
- Struct(target_fields) => {
+ match (source_col.data_type(), target_type) {
+ (_, Struct(target_fields)) => {
Review Comment:
After this patch, runtime support exists for some nested wrappers around structs, but planning is still using the old top-level check
` } else if matches!((&expr_type, &cast_type), (Struct(_), Struct(_))) {`
at datafusion/physical-expr/src/expressions/cast.rs:326
##########
datafusion/common/src/scalar/mod.rs:
##########
@@ -3905,15 +3905,11 @@ impl ScalarValue {
// and recursively casts nested structs. The field name wrapper is arbitrary
// since cast_column only uses the DataType::Struct field definitions inside.
let cast_arr = match target_type {
- DataType::Struct(_) => {
- // Field name is unused; only the struct's inner field names matter
- let target_field = Field::new("_", target_type.clone(), true);
- crate::nested_struct::cast_column(
- &scalar_array,
- &target_field,
- cast_options,
- )?
- }
+ DataType::Struct(_) => crate::nested_struct::cast_column(
Review Comment:
This match DataType::Struct cast entry point and at
datafusion/expr-common/src/columnar_value.rs:317
mean casts such as `List<Struct{...}> -> List<Struct{..., extra}>` and `Dictionary<_, Struct{...}> -> Dictionary<_, Struct{..., extra}>` still fall back to Arrow's generic `cast_with_options`.
I think we should add a shared predicate like "does this target type require name-based nested struct casting?" and reuse it in all three places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] physical-expr-adapter: support casting structs nested inside complex types [datafusion]
Posted by "asubiotto (via GitHub)" <gi...@apache.org>.
asubiotto commented on code in PR #20907:
URL: https://github.com/apache/datafusion/pull/20907#discussion_r2924955640
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -151,7 +156,7 @@ fn cast_struct_column(
///
/// # Arguments
/// * `source_col` - The source array to cast
-/// * `target_field` - The target field definition (including type and metadata)
+/// * `target_type` - The target data type to cast to
Review Comment:
This refactor made sense. `target_field` was not used at all, only its inner `DataType`. There were several callsites that constructed "mock" fields just to be able to call this function with a datatype.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] physical-expr-adapter: support casting structs nested inside complex types [datafusion]
Posted by "asubiotto (via GitHub)" <gi...@apache.org>.
asubiotto commented on PR #20907:
URL: https://github.com/apache/datafusion/pull/20907#issuecomment-4047088120
cc @kosiew since you touched the code most recently. cc @alamb in case this needs to be routed to somebody else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix[physical-expr-adapter]: support casting structs nested inside complex types [datafusion]
Posted by "asubiotto (via GitHub)" <gi...@apache.org>.
asubiotto commented on code in PR #20907:
URL: https://github.com/apache/datafusion/pull/20907#discussion_r2929978701
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -1010,4 +1173,18 @@ mod tests {
assert!(b_col.is_null(0));
assert!(b_col.is_null(1));
}
+
Review Comment:
I initially had parameterized test cases for all types but it was a fair amount of code. I'm not sure, but I'm happy to add them back and we can decide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix[physical-expr-adapter]: support casting structs nested inside complex types [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20907:
URL: https://github.com/apache/datafusion/pull/20907#issuecomment-4106437610
Is this ready to merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix[physical-expr-adapter]: support casting structs nested inside complex types [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #20907:
URL: https://github.com/apache/datafusion/pull/20907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org