You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "LuciferYang (via GitHub)" <gi...@apache.org> on 2026/10/14 04:49:01 UTC
[PR] [common][spark] Fix hilbert sort keys for date, float and double columns [paimon]
LuciferYang opened a new pull request, #10554:
URL: https://github.com/apache/paimon/pull/10554
### Purpose
close #10553
- Spark DATE: `SparkHilbertUDF` cast the date to bigint, which fails analysis under ANSI mode and gives null otherwise. It now takes the days since epoch, the same value `HilbertIndexer` reads for a DATE field.
- FLOAT and DOUBLE: the curve reads only the low 63 bits of each coordinate, so the raw IEEE bits lost the sign and `x` and `-x` shared a curve position. `HilbertIndexer.doubleToOrderedLong` flips the bits the way z-order does and shifts them into the 63-bit domain; the indexer and the Spark UDF both use it.
The other types keep their mapping, so existing hilbert orders on integer columns do not change.
### Tests
- `HilbertIndexerTest#testFloatingPointKeysKeepTheSign`: `x` and `-x` get different keys, and over an 8x8 grid of signed values one coordinate changes sign once along the sorted order.
- `SparkHilbertUDFTest#testDateColumnMapsToEpochDay`: dates from year 1 to 9999 and null map to their epoch day and the null sentinel, with ANSI on and off.
- `CompactProcedureTestBase`: `sys.compact` with `order_strategy => 'hilbert'` on a DATE column under ANSI mode, and on two DOUBLE columns with both signs.
### API and Format
no
### Documentation
no
Generated-by: Claude Opus
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][spark] Fix hilbert sort keys for date, float and double columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10554:
URL: https://github.com/apache/paimon/pull/10554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][spark] Fix hilbert sort keys for date, float and double columns [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10554:
URL: https://github.com/apache/paimon/pull/10554#issuecomment-6205503606
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][spark] Fix hilbert sort keys for date, float and double columns [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #10554:
URL: https://github.com/apache/paimon/pull/10554#issuecomment-6196808391
cc @JingsongLi Could you take a look at this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [common][spark] Fix hilbert sort keys for date, float and double columns [paimon]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #10554:
URL: https://github.com/apache/paimon/pull/10554#issuecomment-6208653777
Thank you @JingsongLi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org