[PR] fix: correct array_append return type and mark as Compatible [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new pull request, #3795:
URL: https://github.com/apache/datafusion-comet/pull/3795

   ## Summary
   
   Two fixes to `CometArrayAppend`:
   
   **Return type mismatch.** The serde was passing `ArrayType(elementType)` (always `containsNull=true`) as the return type to DataFusion. Spark's actual return type is `if (right.nullable) left.dataType.asNullable else left.dataType` — meaning when the element is non-nullable, the input array's `containsNull` flag is preserved. Fixed by passing `expr.dataType` directly, which Spark has already computed correctly.
   
   **Incorrect `Incompatible` classification.** `CometArrayAppend` was marked `Incompatible(None)` with no explanation. The `CaseWhen(IsNotNull(arr), array_append(arr, elem), null)` wrapper already handles the only genuine incompatibility (DataFusion's `array_append` does not preserve null top-level array rows on its own). With the return type fix, the expression now matches Spark's behavior fully. Changed to `Compatible()`.
   
   **Test cleanup.** Removed `spark.comet.expression.ArrayAppend.allowIncompatible=true` from `array_append.sql` since `Compatible` expressions don't need it. Added a comment explaining why `ArrayInsert.allowIncompatible=true` is still needed (on Spark 4.0, `array_append` is a `RuntimeReplaceable` that rewrites to `array_insert(-1)`).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] fix: correct array_append return type and mark as Compatible [datafusion-comet]

["comphead (via GitHub)" <gi...@apache.org>]

comphead merged PR #3795:
URL: https://github.com/apache/datafusion-comet/pull/3795


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org