You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "wiedld (via GitHub)" <gi...@apache.org> on 2025/08/19 22:28:31 UTC
[PR] Document how schema projection works. [datafusion]
wiedld opened a new pull request, #17250:
URL: https://github.com/apache/datafusion/pull/17250
## Which issue does this PR close?
Part of https://github.com/apache/datafusion/issues/12736
## Rationale for this change
Document how schema projection is currently done, and provide guidelines (via docs) of expected behavior to retain in the future.
## What changes are included in this PR?
Docs.
## Are these changes tested?
N/A
## Are there any user-facing changes?
Only docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Document how schema projection works. [datafusion]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on code in PR #17250:
URL: https://github.com/apache/datafusion/pull/17250#discussion_r2286541349
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -55,6 +55,24 @@ use datafusion_common::tree_node::{
/// The rule analyzes the input logical plan, determines the necessary column
/// indices, and then removes any unnecessary columns. It also removes any
/// unnecessary projections from the plan tree.
+///
+/// ## Schema, Field Properties, and Metadata Handling
+///
+/// The `OptimizeProjections` rule preserves schema and field metadata in most optimization scenarios:
+///
+/// **Schema-level metadata preservation by plan type**:
+/// - **Window and Aggregate plans**: Schema metadata is preserved
+/// - **Projection plans**: Schema metadata is preserved per [`projection_schema`](datafusion_expr::logical_plan::projection_schema).
+/// - **Other logical plans**: Schema metadata is preserved unless [`LogicalPlan::recompute_schema`]
+/// is called on plan types that drop metadata
+///
+/// **Field-level properties and metadata**: Individual field properties are preserved when fields
+/// are retained in the optimized plan, determined by [`exprlist_to_fields`](datafusion_expr::utils::exprlist_to_fields)
+/// and [`ExprSchemable::to_field`](datafusion_expr::expr_schema::ExprSchemable::to_field).
+///
+/// **Field precedence**: When the same field appears multiple times, the optimizer
+/// maintains one occurrence and removes duplicates (refer to `RequiredIndices::compact()`),
Review Comment:
Note that `RequiredIndices::compact()` currently [uses sort_unstable](https://github.com/apache/datafusion/blob/4bc069693dc374c1b52c6266ae7a01ef46e1fdf5/datafusion/optimizer/src/optimize_projections/required_indices.rs#L222), which means that it (performantly) picks a winner field -- not the first field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Document how schema projection works. [datafusion]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on PR #17250:
URL: https://github.com/apache/datafusion/pull/17250#issuecomment-3287217915
Thanks for the review @Jefffrey !
Updates are done: https://github.com/apache/datafusion/pull/17250/commits/3443d8dc67443feded4d048735fd686edd862c18
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Document how schema projection works. [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #17250:
URL: https://github.com/apache/datafusion/pull/17250#discussion_r2342968747
##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -2184,14 +2184,22 @@ impl Projection {
/// will be computed.
/// * `exprs`: A slice of `Expr` expressions representing the projection operation to apply.
///
+/// # Metadata Handling
+///
+/// - **Schema-level metadata**: Passed through unchanged from the input schema
+/// - **Field-level metadata**: Determined by each expression via `exprlist_to_fields`, which
+/// calls `Expr::to_field` to handle expression-specific metadata (literals, aliases, etc.)
Review Comment:
```suggestion
/// - **Field-level metadata**: Determined by each expression via [`exprlist_to_fields`], which
/// calls [`Expr::to_field`] to handle expression-specific metadata (literals, aliases, etc.)
```
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -371,6 +371,49 @@ impl ExprSchemable for Expr {
/// Returns a [arrow::datatypes::Field] compatible with this expression.
///
+ /// This function converts an expression into a field with appropriate metadata
+ /// and nullability based on the expression type and context. It is the primary
+ /// mechanism for determining field-level schemas.
+ ///
+ /// # Field Property Resolution
+ ///
+ /// For each expression, the following properties are determined:
+ ///
+ /// ## Data Type Resolution
+ /// - **Column references**: Data type from input schema field
+ /// - **Literals**: Data type inferred from literal value
+ /// - **Aliases**: Data type inherited from the underlying expression (the aliased expression)
+ /// - **Binary expressions**: Result type from type coercion rules
+ /// - **Boolean expressions**: Always a boolean type
+ /// - **Cast expressions**: Target data type from cast operation
+ /// - **Function calls**: Return type based on function signature and argument types
+ ///
+ /// ## Nullability Determination
+ /// - **Column references**: Inherit nullability from input schema field
+ /// - **Literals**: Nullable only if literal value is NULL
+ /// - **Aliases**: Inherit nullability from the underlying expression (the aliased expression)
+ /// - **Binary expressions**: Nullable if either operand is nullable
+ /// - **Boolean expressions**: Always non-nullable (IS NULL, EXISTS, etc.)
+ /// - **Cast expressions**: determined by the input expression's nullability rules
+ /// - **Function calls**: Based on function nullability rules and input nullability
+ ///
+ /// ## Metadata Handling
+ /// - **Column references**: Preserve original field metadata from input schema
+ /// - **Literals**: Use explicitly provided metadata, otherwise empty
+ /// - **Aliases**: Merge underlying expr metadata with alias-specific metadata, preferring the alias metadata
+ /// - **Binary expressions**: field metadata is empty
+ /// - **Boolean expressions**: field metadata is empty
+ /// - **Cast expressions**: determined by the input expression's field metadata handling
+ /// - **Scalar functions**: Generate metadata via function's `return_field_from_args` method,
+ /// with the default implementation returning empty field metadata
+ /// - **Aggregate functions**: Generate metadata via function's `return_field` method,
Review Comment:
```suggestion
/// - **Scalar functions**: Generate metadata via function's [`return_field_from_args`] method,
/// with the default implementation returning empty field metadata
/// - **Aggregate functions**: Generate metadata via function's [`return_field`] method,
```
And add at the bottom of the doc:
```rust
/// [`return_field_from_args`]: crate::ScalarUDF::return_field_from_args
/// [`return_field`]: crate::AggregateUDF::return_field
```
##########
datafusion/expr/src/utils.rs:
##########
@@ -690,7 +690,23 @@ where
err
}
-/// Create field meta-data from an expression, for use in a result set schema
+/// Create schema fields from an expression list, for use in result set schema construction
+///
+/// This function converts a list of expressions into a list of complete schema fields,
+/// making comprehensive determinations about each field's properties including:
+/// - **Data type**: Resolved based on expression type and input schema context
+/// - **Nullability**: Determined by expression-specific nullability rules
+/// - **Metadata**: Computed based on expression type (preserving, merging, or generating new metadata)
+/// - **Table reference scoping**: Establishing proper qualified field references
+///
+/// Each expression is converted to a field by calling `Expr::to_field`, which performs
Review Comment:
```suggestion
/// Each expression is converted to a field by calling [`Expr::to_field`], which performs
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Document how schema projection works. [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #17250:
URL: https://github.com/apache/datafusion/pull/17250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Document how schema projection works. [datafusion]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on code in PR #17250:
URL: https://github.com/apache/datafusion/pull/17250#discussion_r2286506758
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -55,6 +55,24 @@ use datafusion_common::tree_node::{
/// The rule analyzes the input logical plan, determines the necessary column
/// indices, and then removes any unnecessary columns. It also removes any
/// unnecessary projections from the plan tree.
+///
+/// ## Schema, Field Properties, and Metadata Handling
+///
+/// The `OptimizeProjections` rule preserves schema and field metadata in most optimization scenarios:
Review Comment:
This summarizes how the `OptimizeProjections` handles the schema, field props, and metadata.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Document how schema projection works. [datafusion]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on code in PR #17250:
URL: https://github.com/apache/datafusion/pull/17250#discussion_r2286514540
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -55,6 +55,24 @@ use datafusion_common::tree_node::{
/// The rule analyzes the input logical plan, determines the necessary column
/// indices, and then removes any unnecessary columns. It also removes any
/// unnecessary projections from the plan tree.
+///
+/// ## Schema, Field Properties, and Metadata Handling
+///
+/// The `OptimizeProjections` rule preserves schema and field metadata in most optimization scenarios:
Review Comment:
I then added docs to the other major functions used by `OptimizeProjections` (as well as elsewhere):
* `projection_schema`
* `exprlist_to_fields`
* `ExprSchemable::to_field`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org