[I] greatest/least compare struct arguments by field name instead of position when names differ only in case [datafusion-comet]

["viirya (via GitHub)" <gi...@apache.org>]

viirya opened a new issue, #6758:
URL: https://github.com/apache/datafusion-comet/issues/6758

   ### Describe the bug
   
   With `spark.sql.caseSensitive=false`, `greatest` and `least` return a different struct from Spark when the arguments' struct field names differ only in case.
   
   Spark accepts such arguments because `TypeCoercion.haveSameType` compares the types ignoring case and nullability. It then compares the structs field by field by position, and the result keeps the first argument's field names. Comet passes the arguments through DataFusion's `greatest`/`least` coercion. When both structs hold the same set of field names (here `x` and `X`, in different orders), DataFusion's struct coercion matches the fields by name, and Arrow's struct cast reorders the second argument's values by name. So Comet compares different pairs of fields than Spark does.
   
   This affects every supported Spark version (3.4 to 4.1), for float and non-float fields, for structs nested in arrays, and for three or more arguments. With three arguments that also differ in field nullability, the native cast fails instead: `Cannot cast nullable struct field 'X' to non-nullable field`.
   
   ### Steps to reproduce
   
   ```sql
   SET spark.sql.caseSensitive=false;
   SELECT id, greatest(named_struct('x', CAST(id AS DOUBLE), 'X', 1D),
                       named_struct('X', 1D, 'x', CAST(id AS DOUBLE)))
   FROM range(4);
   ```
   
   With `spark.comet.exec.range.enabled=true`, `spark.comet.sparkToColumnar.enabled=true`, and `spark.comet.sparkToColumnar.supportedOperatorList=Range`, Comet runs this natively. For `id = 0` Spark returns `[1.0, 0.0]` and Comet returns `[0.0, 1.0]`. `least` has the same problem: for `id = 2` it returns `[2.0, 1.0]` where Spark returns `[1.0, 2.0]`.
   
   ### Expected behavior
   
   Comet compares struct arguments by position, as Spark does, and returns `[1.0, 0.0]` for `id = 0`.
   
   ### Additional context
   
   `coalesce`, `IF`, `CASE WHEN`, `array`, `map`, `concat`, `map_concat`, `array_append`, `array_insert`, `array_contains`, `array_intersect`, `array_except`, `arrays_overlap`, `=`, `<`, `IN` and `nullif` match Spark for the same arguments. `array_union` with integer struct fields fails natively with `Union is not implemented for 'ListArray'` rather than returning a wrong answer.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] greatest/least compare struct arguments by field name instead of position when names differ only in case [datafusion-comet]

["viirya (via GitHub)" <gi...@apache.org>]

viirya closed issue #6758: greatest/least compare struct arguments by field name instead of position when names differ only in case
URL: https://github.com/apache/datafusion-comet/issues/6758


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org