[I] spark: array_contains compares floats by bits, where Spark treats -0.0 as 0.0 and all NaNs as equal [datafusion]

["zhangfengcdt (via GitHub)" <gi...@apache.org>]

zhangfengcdt opened a new issue, #25981:
URL: https://github.com/apache/datafusion/issues/25981

   ### Describe the bug
   
   `SparkArrayContains` in the `datafusion-spark` crate (`datafusion/spark/src/function/array/array_contains.rs`) delegates the element comparison to `array_has`, which compares values by their bits. Spark's `ArrayContains` compares elements with `genEqual`, under which `-0.0` equals `0.0` and all NaNs are equal, at any depth of a struct or array element. So the two disagree on those values:
   
   | | Spark | `SparkArrayContains` |
   | --- | --- | --- |
   | `array_contains(array(-0.0D), 0.0D)` | `true` | `false` |
   | `array_contains(array(double('NaN')), double('NaN'))` | `true` | `true` only when the NaN bit patterns match |
   
   The null semantics patched in on top of `array_has` are right; only the equality differs.
   
   ### To Reproduce
   
   ```rust
   use std::sync::Arc;
   
   use arrow::array::{Float64Array, ListArray};
   use arrow::datatypes::{DataType, Field};
   use datafusion_common::ScalarValue;
   use datafusion_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl};
   use datafusion_spark::function::array::array_contains::SparkArrayContains;
   
   let array = ListArray::new(
       Arc::new(Field::new("item", DataType::Float64, true)),
       arrow::buffer::OffsetBuffer::new(vec![0, 1].into()),
       Arc::new(Float64Array::from(vec![-0.0])),
       None,
   );
   let result = SparkArrayContains::default().invoke_with_args(ScalarFunctionArgs {
       args: vec![
           ColumnarValue::Array(Arc::new(array)),
           ColumnarValue::Scalar(ScalarValue::Float64(Some(0.0))),
       ],
       arg_fields: vec![],
       number_rows: 1,
       return_field: Arc::new(Field::new("result", DataType::Boolean, true)),
       config_options: Default::default(),
   })?;
   // Spark returns true for array_contains(array(-0.0D), 0.0D); this returns false.
   ```
   
   ### Expected behavior
   
   `array_contains` matches Spark's `genEqual` for float and double elements: `-0.0` equals `0.0`, and any NaN equals any other NaN, including inside struct and array elements.
   
   ### Additional context
   
   Found while working on apache/datafusion-comet#6520. Comet currently routes `array_contains` on float elements away from this function because of the mismatch, and is adding its own kernel with Spark's equality in the meantime; a fix here would let it drop that kernel. `arrays_overlap` (#20781) in the same crate likely has the same equality.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org