[I] make_array: relax element-type equality to accept inputs differing only in nested-field nullability [datafusion]

["schenksj (via GitHub)" <gi...@apache.org>]

schenksj opened a new issue, #22366:
URL: https://github.com/apache/datafusion/issues/22366

   ## Summary
   
   `make_array` (in `datafusion-functions-nested`) panics when called with arrays whose element types share the same shape but differ in nested-field nullability. Spark, Postgres, and `arrow::compute::concat` all accept this and widen `nullable` to `true` in the result type. DataFusion's `make_array_inner` is stricter, which propagates up to any caller that builds `array(...)` over heterogeneously-produced child expressions.
   
   ## Repro symptom
   
   Real-world surfacing in [apache/datafusion-comet](https://github.com/apache/datafusion-comet) on a Delta Lake CDF write that builds `array(struct(id, b, _change_type=lit(\"delete\")), struct(id, b, _change_type=col(...)))` — one arm's `_change_type` is `Utf8` non-nullable (from a literal), another is `Utf8` nullable:
   
   ```
   panicked at arrow-data-58.2.0/src/transform/mod.rs:422:
   assertion `left == right` failed: Arrays with inconsistent types passed to MutableArrayData
    left: Struct([Field { name: \"id\", data_type: Int64, nullable: true },
                  Field { name: \"b\",  data_type: Int32 },
                  Field { name: \"_change_type\", data_type: Utf8 }])
   right: Struct([Field { name: \"id\", data_type: Int64, nullable: true },
                  Field { name: \"b\",  data_type: Int32 },
                  Field { name: \"_change_type\", data_type: Utf8, nullable: true }])
   ```
   
   Stack: `make_array_inner` → `MutableArrayData::with_capacities`.
   
   ## Proposal
   
   `make_array` should accept element types that are equal under nullability-widening (recursively, for nested structs/lists/maps). Concretely:
   
   - Compute the merged element type by walking each child's `DataType` and OR-ing the `nullable` flag at every level (this is essentially `Field::try_merge` minus the type-promotion arm).
   - Cast each child to the merged type before handing to `MutableArrayData`.
   - Return `ArrayType` with `containsNull = true` if any merge raised a nullability flag.
   
   This matches what `coerce_types`-style coercion does elsewhere in the planner, but applied at execution time when input arrays still disagree (the planner can't always normalize, e.g. when the array is built from disjoint sources like Delta CDF struct literals).
   
   ## Why this matters
   
   It blocks native execution of any plan that produces struct elements from multiple sources (CDF writes, UNION ALL inside an `array()`, manually-constructed plans bypassing TypeCoercion). Workaround today: callers must insert explicit casts upstream, or fall back to a non-DataFusion evaluator — both of which lose perf.
   
   ## Related caller-side mitigation (for context)
   
   Comet just landed a serde-side decline in [4cb9b4dc](https://github.com/apache/datafusion-comet/commit/) that falls back to Spark's JVM evaluator when `CreateArray`'s children have different `DataType`s. That fix is conservative but loses native execution. Upstreaming the relaxation here would let downstream projects keep native execution and would help any other Arrow-based engine hitting the same shape.
   
   I can put up a PR if the approach lands well.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] make_array: relax element-type equality to accept inputs differing only in nested-field nullability [datafusion]

["schenksj (via GitHub)" <gi...@apache.org>]

schenksj commented on issue #22366:
URL: https://github.com/apache/datafusion/issues/22366#issuecomment-4581492644

   Update from the Comet side, with a finding that narrows the scope of this.
   
   **Empirical asymmetry in `MutableArrayData`:** while implementing the caller-side mitigation I found that `make_array` already tolerates **container** nullability differences — an `ArrayType.containsNull` / `MapType.valueContainsNull` mismatch is coerced and runs fine (e.g. `array(array(int) <non-null elem>, array(int) <nullable elem>)`). The panic fires **only** on a `StructType` field's nullability mismatch. So the strictness is really in the struct branch of `with_capacities`, not the list/map branches. The OR-merge-at-every-level proposal above is a clean superset and still correct, but the minimal fix may only need to touch struct field merging.
   
   **Corrected caller-side reference:** the "Related caller-side mitigation" note links a placeholder commit. The actual Comet fix is apache/datafusion-comet#4533. It declines (falls back to Spark) **only** when the children still differ after normalizing container nullability — i.e. exactly the struct-field-nullability case — so it does not over-decline the container-only arrays that already work natively. It's still conservative (loses native execution for the struct case); upstreaming the relaxation here would remove the need for it.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org