You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "linhr (via GitHub)" <gi...@apache.org> on 2026/07/13 12:45:29 UTC
[I] Missing field metadata for projections during physical planning [datafusion]
linhr opened a new issue, #23529:
URL: https://github.com/apache/datafusion/issues/23529
### Describe the bug
DataFusion seems to drop field metadata for projections during physical planning. This may cause schema mismatch when the physical schema is later validated against the logical schema (e.g., when `execution.skip_physical_aggregate_schema_check` is `false`).
### To Reproduce
Here is a simple test case to show this behavior.
```rust
#[cfg(test)]
mod tests {
use std::collections::HashMap;
use std::sync::Arc;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::common::ToDFSchema;
use datafusion::common::metadata::FieldMetadata;
use datafusion::functions_aggregate::min_max::max_udaf;
use datafusion::logical_expr::expr::AggregateFunction;
use datafusion::logical_expr::{EmptyRelation, Expr, LogicalPlan};
use datafusion::physical_planner::{DefaultPhysicalPlanner, PhysicalPlanner};
use datafusion::prelude::{SessionContext, col};
use datafusion_expr::logical_plan::{Aggregate, Projection};
#[tokio::test]
async fn test_aggregate_planning_fails_on_projection_with_field_metadata() {
let schema = Schema::new(vec![Field::new("value", DataType::Utf8, false)]);
let input = LogicalPlan::EmptyRelation(EmptyRelation {
produce_one_row: false,
schema: Arc::new(schema.to_dfschema().unwrap()),
});
let metadata = FieldMetadata::from(HashMap::from([("foo".to_string(), "bar".to_string())]));
let projection = LogicalPlan::Projection(
Projection::try_new(
vec![col("value").alias_with_metadata("value", Some(metadata))],
Arc::new(input),
)
.unwrap(),
);
let aggregate = LogicalPlan::Aggregate(
Aggregate::try_new(
Arc::new(projection),
vec![],
vec![Expr::AggregateFunction(AggregateFunction::new_udf(
max_udaf(),
vec![col("value")],
false,
None,
vec![],
None,
))],
)
.unwrap(),
);
DefaultPhysicalPlanner::default()
.create_physical_plan(&aggregate, &SessionContext::new().state())
.await
.unwrap();
}
}
```
The test would fail with the following error:
```
called `Result::unwrap()` on an `Err` value: Internal("Physical input schema should be the same as the one converted from logical input schema. Differences: \n\t- field metadata at index 0 [value]: (physical) {} vs (logical) {\"foo\": \"bar\"}")
```
### Expected behavior
`DefaultPhysicalPlanner` should handle field metadata in the logical plan, possibly inside `create_project_physical_exec_with_props`.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Missing field metadata for projections during physical planning [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #23529:
URL: https://github.com/apache/datafusion/issues/23529#issuecomment-5115396183
untake
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Missing field metadata for projections during physical planning [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on issue #23529:
URL: https://github.com/apache/datafusion/issues/23529#issuecomment-4968890459
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Missing field metadata for projections during physical planning [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew closed issue #23529: Missing field metadata for projections during physical planning
URL: https://github.com/apache/datafusion/issues/23529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org