[I] Nested floating-point IN membership does not match Spark for signed zero [datafusion-comet]

["sunchao (via GitHub)" <gi...@apache.org>]

sunchao opened a new issue, #6019:
URL: https://github.com/apache/datafusion-comet/issues/6019

   ### Describe the bug
   
   `IN` / `InSet` floating-point normalization added in #5472 applies only to top-level
   `FLOAT` and `DOUBLE` operands. `ARRAY<DOUBLE>` and structs containing floating-point
   leaves fall through unnormalized.
   
   Spark evaluates `IN` with `TypeUtils.getInterpretedOrdering`, which treats `-0.0` and
   `0.0` as equal at floating-point leaves. The native nested membership comparator can
   distinguish those bit patterns, so Comet can silently return `false` where Spark returns
   `true`.
   
   ### Steps to reproduce
   
   Use a Parquet table so the signed-zero bits survive input materialization:
   
   ```sql
   CREATE TABLE comet_nested_in_zero (
     id INT,
     a ARRAY<DOUBLE>,
     s STRUCT<v: DOUBLE>
   ) USING parquet;
   
   INSERT INTO comet_nested_in_zero VALUES (
     1,
     array(CAST('-0.0' AS DOUBLE)),
     named_struct('v', CAST('-0.0' AS DOUBLE))
   );
   
   SELECT
     id,
     a IN (array(CAST('0.0' AS DOUBLE))) AS array_match,
     s IN (named_struct('v', CAST('0.0' AS DOUBLE))) AS struct_match
   FROM comet_nested_in_zero;
   ```
   
   Compare with `spark.comet.enabled=false`, and require the native membership path in the
   Comet run so fallback cannot hide the mismatch. Repeat for `FLOAT`, `InSet`, and a
   column candidate such as `a IN (b)`.
   
   ### Expected behavior
   
   Nested floating-point leaves use Spark-compatible membership equality: `-0.0` and `0.0`
   match. NaN representations should also be checked and either normalized consistently or
   routed through Spark fallback.
   
   ### Additional context
   
   - #5472 fixes scalar `FLOAT` / `DOUBLE` comparison and membership paths, but deliberately
     does not recurse into nested types.
   - #5507 tracks nested `ORDER BY` and window-rank semantics, not `IN`.
   - #5191 covered `arrays_overlap` / `array_position` and is now closed.
   - This is a pre-existing limitation, not a regression introduced by #5472.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Nested floating-point IN membership does not match Spark for signed zero [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #6019: Nested floating-point IN membership does not match Spark for signed zero
URL: https://github.com/apache/datafusion-comet/issues/6019


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Nested floating-point IN membership does not match Spark for signed zero [datafusion-comet]

["mizulun (via GitHub)" <gi...@apache.org>]

mizulun commented on issue #6019:
URL: https://github.com/apache/datafusion-comet/issues/6019#issuecomment-5741940718

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org