You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "sunchao (via GitHub)" <gi...@apache.org> on 2026/08/27 17:25:54 UTC
[I] Narrow strict floating-point admission for corrected scalar sort keys [datafusion-comet]
sunchao opened a new issue, #5506:
URL: https://github.com/apache/datafusion-comet/issues/5506
### What is the problem the feature request solves?
After #5469, native Sort, Window, WindowGroupLimit, and range-partition comparison keys normalize scalar `FLOAT` and `DOUBLE` NaNs and signed zeros consistently with Spark. Admission is still more conservative: `CometSortOrder.getSupportLevel` calls the shared `strictFloatingPointReason` helper, which rejects both scalar and nested floating types when `spark.comet.exec.strictFloatingPoint=true`.
Strict-mode users therefore still lose native scalar floating sorts even though #5469 fixes their comparison semantics. This follow-up should evaluate and test narrowing that specific admission decision, without removing the remaining nested-type protections or changing other callers of the shared helper.
### Describe the potential solution
Use a Spark session with Comet enabled and `CometShuffleManager` configured. Create a Parquet input so constant folding cannot remove the sort:
```sql
CREATE TABLE comet_scalar_sort_case (id INT, v DOUBLE) USING parquet;
INSERT INTO comet_scalar_sort_case VALUES
(1, CAST('-0.0' AS DOUBLE)), (2, CAST('0.0' AS DOUBLE)), (3, 1.0);
SET spark.sql.adaptive.enabled=false;
SET spark.sql.shuffle.partitions=1;
SET spark.comet.exec.strictFloatingPoint=true;
EXPLAIN FORMATTED
SELECT id, v FROM comet_scalar_sort_case ORDER BY v, id DESC;
SET spark.comet.exec.strictFloatingPoint=false;
EXPLAIN FORMATTED
SELECT id, v FROM comet_scalar_sort_case ORDER BY v, id DESC;
```
Require the strict-mode plan to use native `CometSortExec` only after its compatibility is established; the current strict-mode plan falls back. Check results against Comet disabled, not only against the non-strict native path. The zero peers must be ordered by the secondary key (`id` 2 before 1), and returned zero signs must remain unchanged.
The acceptance tests should cover `FLOAT` and `DOUBLE`, both sort directions and null orders, compound keys, signed zeros, and independently constructed positive/negative NaN payloads. Cover ordinary window ordering and `RANK`/`DENSE_RANK` cutoffs with actual native operator assertions. Retain strict-mode fallback controls for arrays/structs containing floats, and test all supported Spark versions before narrowing the policy.
### Additional context
The current [SortOrder gate](https://github.com/apache/datafusion-comet/blob/96eafdfe14384976cb18b7364c4cf84d7d1fb723/spark/src/main/scala/org/apache/comet/serde/CometSortOrder.scala#L33-L39) delegates to a [recursive shared policy](https://github.com/apache/datafusion-comet/blob/96eafdfe14384976cb18b7364c4cf84d7d1fb723/spark/src/main/scala/org/apache/comet/serde/SupportLevel.scala#L87-L104). The native [scalar key normalization](https://github.com/apache/datafusion-comet/blob/96eafdfe14384976cb18b7364c4cf84d7d1fb723/native/core/src/execution/planner.rs#L927-L963) is intentionally narrower. Keep this admission follow-up separate from #5468's scalar wrong-result fix and from the nested ordering/rank limitation. #2626 is closed and does not track this remaining admission work.
A fresh Spark 4.1.3 planning probe confirmed native scalar sorts with strict mode disabled and Spark sorts with it enabled for both FLOAT and DOUBLE. The reused JVM build's `CometSortOrder`, `SupportLevel`, and `CometWindowExec` sources are byte-identical to the reviewed #5469 head. This checks admission; it does not claim a new full-binary runtime validation of the scalar normalization fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Narrow strict floating-point admission for corrected scalar sort keys [datafusion-comet]
Posted by "0lai0 (via GitHub)" <gi...@apache.org>.
0lai0 commented on issue #5506:
URL: https://github.com/apache/datafusion-comet/issues/5506#issuecomment-5654398804
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Narrow strict floating-point admission for corrected scalar sort keys [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #5506: Narrow strict floating-point admission for corrected scalar sort keys
URL: https://github.com/apache/datafusion-comet/issues/5506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org