You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2025/08/22 03:26:46 UTC
[PR] Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
kosiew opened a new pull request, #17281:
URL: https://github.com/apache/datafusion/pull/17281
## Which issue does this PR close?
* Closes #16579
## Rationale for this change
Evolving data sources often have structural mismatches with the expected table schema, especially when nested `Struct` types are involved. This PR introduces robust handling for schema adaptation and column casting within Apache DataFusion to ensure compatibility and correctness when processing such evolving schemas.
## What changes are included in this PR?
* Introduces `cast_column` for recursively casting nested `StructArray` fields to match target schema.
* Adds compatibility checks to prevent casting nullable fields to non-nullable targets.
* Updates the `SchemaAdapter` and `SchemaMapping` logic to leverage `cast_column`.
* Adds thorough unit tests covering:
* Casting structs with reordering, extra, and missing fields
* Preserving parent nullability
* Structs containing arrays and maps
* Schema mapping and record batch transformation
* Fixes column casting behavior in `pruning_predicate` by using `cast_column` instead of generic Arrow cast.
* Updates documentation:
* Adds new guide: `docs/source/library-user-guide/schema_adapter.md`
* References this guide in main user and API docs
## Are these changes tested?
Yes, extensive tests are included that:
* Validate the `cast_column` logic across multiple complex nested schemas.
* Verify compatibility validation logic.
* Test end-to-end behavior of `SchemaAdapter::map_batch()` with various structural transformations.
* Ensure correct pruning predicate behavior when stats use structs with different field types.
## Are there any user-facing changes?
Yes:
* Users benefit from improved compatibility when reading nested structured data with evolving schemas.
* Documentation has been expanded to include a new section explaining how schema adaptation works and how to use `cast_column`.
There are no breaking changes to public APIs.
---
This change enhances DataFusion's resilience to schema drift and paves the way for more robust handling of semi-structured data. ✨
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] DRAFT - Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#issuecomment-3270960175
> @alamb ,
>
> > is there any way to break this PR into smaller PRs? It is very challenging to review large PRs (as it requires a large amount of contiguous time).
>
> Certainly. I started with #17468
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] DRAFT Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#discussion_r2303007475
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -159,14 +200,23 @@ pub fn cast_column(source_col: &ArrayRef, target_field: &Field) -> Result<ArrayR
pub fn validate_struct_compatibility(
source_fields: &[FieldRef],
target_fields: &[FieldRef],
-) -> Result<bool> {
+) -> Result<()> {
// Check compatibility for each target field
for target_field in target_fields {
// Look for matching field in source by name
if let Some(source_field) = source_fields
.iter()
.find(|f| f.name() == target_field.name())
{
+ // Ensure nullability is compatible. It is invalid to cast a nullable
+ // source field to a non-nullable target field as this may discard
+ // null values.
Review Comment:
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#issuecomment-3223011923
@adriangb ,
> integrate this into PhysicalExprAdapter (https://github.com/apache/datafusion/tree/main/datafusion/physical-expr-adapter )?
Yes, I can, after all the kinks in this PR is ironed out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#issuecomment-3261770095
@kosiew -- is there any way to break this PR into smaller PRs? It is very challenging to review large PRs (as it requires a large amount of contiguous time).
I think @adriangb was also hinting at something similar in https://github.com/apache/datafusion/pull/17281#discussion_r2301025273
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#discussion_r2303202853
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -210,9 +210,22 @@ impl ColumnarValue {
) -> Result<ColumnarValue> {
let cast_options = cast_options.cloned().unwrap_or(DEFAULT_CAST_OPTIONS);
match self {
- ColumnarValue::Array(array) => Ok(ColumnarValue::Array(
- kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
- )),
+ ColumnarValue::Array(array) => match cast_type {
+ // fix https://github.com/apache/datafusion/issues/17285
+ DataType::Struct(_) => {
+ let field = Field::new("", cast_type.clone(), true);
+ Ok(ColumnarValue::Array(cast_column(
+ array,
+ &field,
+ &cast_options,
+ )?))
+ }
+ _ => Ok(ColumnarValue::Array(kernels::cast::cast_with_options(
+ array,
+ cast_type,
+ &cast_options,
+ )?)),
+ },
Review Comment:
Removed this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] DRAFT Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#discussion_r2300933932
##########
datafusion/common/src/nested_struct.rs:
##########
@@ -159,14 +200,23 @@ pub fn cast_column(source_col: &ArrayRef, target_field: &Field) -> Result<ArrayR
pub fn validate_struct_compatibility(
source_fields: &[FieldRef],
target_fields: &[FieldRef],
-) -> Result<bool> {
+) -> Result<()> {
// Check compatibility for each target field
for target_field in target_fields {
// Look for matching field in source by name
if let Some(source_field) = source_fields
.iter()
.find(|f| f.name() == target_field.name())
{
+ // Ensure nullability is compatible. It is invalid to cast a nullable
+ // source field to a non-nullable target field as this may discard
+ // null values.
Review Comment:
I assume this means that you can't cast null values into a non-nullable schema. Is that right?
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -210,9 +210,22 @@ impl ColumnarValue {
) -> Result<ColumnarValue> {
let cast_options = cast_options.cloned().unwrap_or(DEFAULT_CAST_OPTIONS);
match self {
- ColumnarValue::Array(array) => Ok(ColumnarValue::Array(
- kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
- )),
+ ColumnarValue::Array(array) => match cast_type {
+ // fix https://github.com/apache/datafusion/issues/17285
+ DataType::Struct(_) => {
+ let field = Field::new("", cast_type.clone(), true);
+ Ok(ColumnarValue::Array(cast_column(
+ array,
+ &field,
+ &cast_options,
+ )?))
+ }
+ _ => Ok(ColumnarValue::Array(kernels::cast::cast_with_options(
+ array,
+ cast_type,
+ &cast_options,
+ )?)),
+ },
Review Comment:
Nice! Would it be possible to isolate this fix into it's own PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#issuecomment-3264831200
@alamb ,
> is there any way to break this PR into smaller PRs? It is very challenging to review large PRs (as it requires a large amount of contiguous time).
Certainly.
I started with #17468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] DRAFT Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#issuecomment-3223795056
> @adriangb ,
>
> > integrate this into PhysicalExprAdapter (https://github.com/apache/datafusion/tree/main/datafusion/physical-expr-adapter )?
>
> Yes, I can, after all the kinks in this PR is ironed out.
Thank you I will try to review this PR today
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] DRAFT Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#discussion_r2303006583
##########
datafusion/expr-common/src/columnar_value.rs:
##########
@@ -210,9 +210,22 @@ impl ColumnarValue {
) -> Result<ColumnarValue> {
let cast_options = cast_options.cloned().unwrap_or(DEFAULT_CAST_OPTIONS);
match self {
- ColumnarValue::Array(array) => Ok(ColumnarValue::Array(
- kernels::cast::cast_with_options(array, cast_type, &cast_options)?,
- )),
+ ColumnarValue::Array(array) => match cast_type {
+ // fix https://github.com/apache/datafusion/issues/17285
+ DataType::Struct(_) => {
+ let field = Field::new("", cast_type.clone(), true);
+ Ok(ColumnarValue::Array(cast_column(
+ array,
+ &field,
+ &cast_options,
+ )?))
+ }
+ _ => Ok(ColumnarValue::Array(kernels::cast::cast_with_options(
+ array,
+ cast_type,
+ &cast_options,
+ )?)),
+ },
Review Comment:
I agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#issuecomment-3215032372
I unfortunately will be out for the weekend and can't review until next week but marked myself for review and will look then. @kosiew are you able to integrate this into `PhysicalExprAdapter` (https://github.com/apache/datafusion/tree/main/datafusion/physical-expr-adapter )? Since we plan on deprecating `SchemaAdatper` in favor of `PhysicalExprAdapter` it's important that the latter can handle the same things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] DRAFT - Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17281:
URL: https://github.com/apache/datafusion/pull/17281#issuecomment-3578538454
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] DRAFT - Expose and generalize cast_column to enable struct → struct casting in more contexts [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #17281: DRAFT - Expose and generalize cast_column to enable struct → struct casting in more contexts
URL: https://github.com/apache/datafusion/pull/17281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org