[I] Internal error: CAST over an expression simplified to a metadata-carrying column, under UNION ALL + aggregate (physical vs logical field metadata) [datafusion]

["TRO-Wolf (via GitHub)" <gi...@apache.org>]

TRO-Wolf opened a new issue, #26178:
URL: https://github.com/apache/datafusion/issues/26178

   ### Describe the bug
   
   A `CAST` over an expression that the optimizer simplifies to a bare column carrying field metadata raises an internal error when it sits in the branches of a `UNION ALL` under an aggregate:
   
   ```
   Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences:
   	- field metadata at index 0 [x]: (physical) {"PARQUET:field_id": "1"} vs (logical) {}.
   ```
   
   Reproduced on DataFusion 54.1.0 (through `datafusion-python` 54.1.0, Linux x86_64). Not checked against `main`.
   
   ### To Reproduce
   
   ```python
   import datafusion, pyarrow as pa
   
   schema = pa.schema([
       pa.field("b", pa.int64(), nullable=False, metadata={"PARQUET:field_id": "1"}),
       pa.field("n", pa.int64(), metadata={"PARQUET:field_id": "2"}),
   ])
   t = pa.table({"b": [1, 2, 3], "n": [1, None, 3]}, schema=schema)
   
   ctx = datafusion.SessionContext()
   ctx.register_record_batches("t", [t.to_batches()])
   
   ctx.sql("""
       SELECT max(x) AS m, count(x) AS c FROM
         (SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t
          UNION ALL
          SELECT CAST(coalesce(b, 7) AS BIGINT) AS x FROM t) q
   """).collect()
   ```
   
   `b` is `NOT NULL`, so `coalesce(b, 7)` simplifies to `b`.
   
   Controls on the same build, all of which answer:
   
   | Query | Result |
   |---|---|
   | the same query without the `UNION ALL` | `m = 3, c = 3` |
   | `UNION ALL` over `CAST(b AS BIGINT)` (no `coalesce`) | `m = 3, c = 6` |
   | the failing query against the same data registered without field metadata | `m = 3, c = 6` |
   
   ### Expected behavior
   
   `m = 3, c = 6`.
   
   ### Additional context
   
   Reading the code, the logical side (`cast_output_field`) and the physical side (`CastExpr`'s resolved target field) both propagate the child's field for a default-target cast, so they agree when both are derived from the same plan state. Here they are not: the `UNION`'s schema was computed before the `coalesce` was simplified away (child = a function result, no metadata), while the physical plan is built from the simplified branches (child = the column, with its metadata).
   
   Two possible directions, offered as a suggestion only:
   
   1. Recompute the union schema after expression simplification. This fixes the `UNION` shape.
   2. Treat a cast's output as a new value that carries none of its child's field metadata, on both the logical and the physical side. This would fix every boundary whose schema is fixed before simplification, and it also removes a visible oddity: today `SELECT CAST(col AS <type>)` returns a field that still carries the source column's `PARQUET:field_id`.
   
   Possibly related: #19049 (closed; `UNION ALL` plus `optimize_projections` with Parquet field metadata), #25978 and #25369 (logical/physical mismatch from a rewrite under an aggregate).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org