[I] [Bug] Hilbert sort fails on Spark DATE columns and ignores the sign of FLOAT and DOUBLE values [paimon]

["LuciferYang (via GitHub)" <gi...@apache.org>]

LuciferYang opened a new issue, #10553:
URL: https://github.com/apache/paimon/issues/10553

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   ### Paimon version
   
   master (726675eac7)
   
   ### Compute Engine
   
   Spark (DATE, FLOAT, DOUBLE) and Flink (FLOAT, DOUBLE)
   
   ### Minimal reproduce step
   
   DATE column on Spark:
   
   ```sql
   CREATE TABLE T (id INT, dt DATE) TBLPROPERTIES ('bucket' = '-1');
   INSERT INTO T VALUES (1, DATE '2024-01-03'), (2, DATE '1969-12-31');
   CALL sys.compact(table => 'T', order_strategy => 'hilbert', order_by => 'dt,id');
   ```
   
   DOUBLE columns, either engine: hilbert-sort a table whose `x, y` columns hold both negative and positive values.
   
   ### What doesn't meet your expectations?
   
   DATE on Spark: `SparkHilbertUDF` casts the date to bigint. With ANSI mode on, the default in Spark 4, the compact fails at analysis with `DATATYPE_MISMATCH.CAST_WITH_FUNC_SUGGESTION: cannot cast "DATE" to "BIGINT"`. With ANSI off the cast yields null for every row, so the date column does not take part in the clustering.
   
   FLOAT and DOUBLE: both `HilbertIndexer` and `SparkHilbertUDF` pass `Double.doubleToLongBits` to a curve built with `HilbertCurve.bits(63)`, which reads only the low 63 bits of each coordinate. The sign bit is dropped, so `x` and `-x` get the same curve position, and points from both signs of the plane end up interleaved in the sorted output.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] Hilbert sort fails on Spark DATE columns and ignores the sign of FLOAT and DOUBLE values [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi closed issue #10553: [Bug] Hilbert sort fails on Spark DATE columns and ignores the sign of FLOAT and DOUBLE values
URL: https://github.com/apache/paimon/issues/10553


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org