You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "jackylee-ch (via GitHub)" <gi...@apache.org> on 2026/09/26 08:44:13 UTC
[PR] [spark] Support decimal columns in zorder clustering [paimon]
jackylee-ch opened a new pull request, #10196:
URL: https://github.com/apache/paimon/pull/10196
## Purpose
`SparkZOrderUDF.sortedLexicographically` dispatches on the column type and
throws `Cannot use column ... of type ... in ZOrdering, the type is
unsupported` for anything it does not list. `DecimalType` is not listed, so
`compact(..., order_strategy => 'zorder', order_by => '<decimal>')` hard-fails
on a decimal column, while `SparkHilbertUDF` — the sibling clustering strategy
in the same package — already clusters decimals. The two strategies disagree
on a common type.
## Change
Handle `DecimalType` in the z-order dispatch the same way Hilbert does, and the
way z-order already handles timestamp/date: cast to long and order that. The
long key is a coarse clustering hint, which is all z-order needs.
## Tests
`SparkZOrderUDFTest.testDecimalColumnIsSupported` z-orders a decimal column
with a null and asserts the null maps to the zero sentinel while the non-null
z-values rise with the value. It throws "the type is unsupported" without the
change.
Written with Claude Code; verification is mine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support decimal columns in zorder clustering [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10196:
URL: https://github.com/apache/paimon/pull/10196#issuecomment-5950997990
[P2] Handle decimals outside BIGINT range in the new z-order path
The new branch at `SparkZOrderUDF.java:368-369` accepts every `DecimalType`, but casting it to `LongType` makes the support depend on both the value and Spark's ANSI setting. Legal `DECIMAL(38,2)` values such as `9223372036854775808.00` and `-9223372036854775809.00` raise `CAST_OVERFLOW` when ANSI is enabled. This is also the default setting in Spark 4.
I reproduced this through an actual persisted Paimon table, rather than invoking the UDF alone:
```sql
SET spark.sql.ansi.enabled=true;
CREATE TABLE T (id INT, amount DECIMAL(38,2)) TBLPROPERTIES ('bucket'='-1');
INSERT INTO T VALUES (1, 9223372036854775808.00), (2, -9223372036854775809.00);
CALL paimon.sys.compact(table => 'test.T', order_strategy => 'zorder', order_by => 'amount');
```
The compact job aborts in the decimal-to-BIGINT cast. The old snapshot remains readable and no replacement snapshot is published, so I did not observe data loss, but the advertised decimal compaction cannot complete for these valid values. Ordinary positive/negative fractional values and nulls do compact and retain their original data.
Please use an encoding that covers the decimal domain without an ANSI-sensitive narrowing cast, and add a procedure-level regression test with out-of-BIGINT values. The existing decimal encoding in `ZIndexer` is a useful reference. Mirroring Hilbert's current cast preserves its older limitation; it does not cover the full domain here.
Validation: all three existing `SparkZOrderUDFTest` cases pass; an additional actual-table SQL probe verifies successful in-range compaction and reproduces the out-of-range failure on Spark 3.5.8 and Spark 4.1.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support decimal columns in zorder clustering [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10196:
URL: https://github.com/apache/paimon/pull/10196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support decimal columns in zorder clustering [paimon]
Posted by "jackylee-ch (via GitHub)" <gi...@apache.org>.
jackylee-ch commented on PR #10196:
URL: https://github.com/apache/paimon/pull/10196#issuecomment-5954753545
Fixed in 5a40eb65c. The decimal branch no longer casts to `LongType` (which raises `CAST_OVERFLOW` under ANSI — Spark 4's default — for values outside the BIGINT range and aborts the compact). It now encodes the unscaled value as a fixed-width, sign-flipped two's-complement key, the same scheme the core `ZIndexer` uses, so the full decimal domain is covered at order-preserving resolution.
Verified locally in `SparkZOrderUDFTest` (4 passed): updated the DECIMAL(10,2) case and added a DECIMAL(38,2) case with `spark.sql.ansi.enabled=true` and values just past both Long.MAX and Long.MIN, asserting the z-value is strictly increasing with the decimal. Reverting to the long cast makes that case fail with `[CAST_OVERFLOW] ... cannot be cast to "BIGINT"` — your exact repro. spotless and `git diff --check` clean. The procedure-level (`CALL compact ... zorder`) smoke runs in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Support decimal columns in zorder clustering [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10196:
URL: https://github.com/apache/paimon/pull/10196#issuecomment-5967539206
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org