[I] Logical schema of an aggregate over GROUPING SETS is more nullable than the physical schema [datafusion]

["alexandrefimov (via GitHub)" <gi...@apache.org>]

alexandrefimov opened a new issue, #24968:
URL: https://github.com/apache/datafusion/issues/24968

   ### Describe the bug
   
   `Aggregate::try_new_with_schema` marks every grouping expression nullable whenever a grouping set is present (`datafusion/expr/src/logical_plan/plan.rs:4102-4113`), so an aggregate's logical schema says nullable where the physical schema says not-null.
   
   Measured on `3266eaa91` over `t(c utf8 NOT NULL, a i64 NOT NULL)`, where `?` marks a nullable field:
   
   ```
   SELECT c, a, count(*) FROM t GROUP BY GROUPING SETS ((c,a),(c))
     logical : c?, a?, count(*)
     physical: c,  a?, count(*)
   
   SELECT c, a, count(*) FROM t GROUP BY GROUPING SETS ((c,a))
     logical : c?, a?, count(*)
     physical: c,  a,  count(*)
   ```
   
   `c` belongs to every set of the first query, and the second query has a single set, so no row is padded with a null in either column.
   
   The physical side computes this per expression in `PhysicalGroupBy::group_fields` (`datafusion/physical-plan/src/aggregates/mod.rs:576-590`) as `group_expr_nullable || expr.nullable(input_schema)?` — set-absence on one side, the expression's own nullability on the other. That is the rule it has used since #12256, which changed `physical-plan` and left the logical side as it was.
   
   ### Expected behavior
   
   A grouping expression is nullable in the logical schema when the input makes it nullable, or when some grouping set leaves it out.
   
   Before writing that, I would like to know what you expect it to move. `DataFrame::schema()` narrows for these queries, so optimizer rules keyed on nullability, EXPLAIN snapshots and sqllogictest results can shift with it, and crates outside the repo may read the wider schema today. If it is a change you want, I am glad to open the PR.
   
   ### Additional context
   
   Found while comparing relation-level schema derivation across Substrait implementations: for a Substrait plan with grouping sets DataFusion answers with the logical schema, so a consumer that compares schemas sees the wider one while execution produces the narrower.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org