You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/09/20 13:17:57 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-6049-1c25b492bfb5d4615c7838cf24ec4da867263034 created (now d1bf687ebb)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6049-1c25b492bfb5d4615c7838cf24ec4da867263034
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at d1bf687ebb docs: correct the stale range-partitioning strict floating-point rule (#6049)
This branch includes the following new commits:
new d1bf687ebb docs: correct the stale range-partitioning strict floating-point rule (#6049)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: docs: correct the stale range-partitioning strict floating-point rule (#6049)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6049-1c25b492bfb5d4615c7838cf24ec4da867263034
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit d1bf687ebb13c1439057d0e7e2d15286f233c3d0
Author: Andy Grove <ag...@apache.org>
AuthorDate: Sun Sep 20 13:17:35 2026 +0000
docs: correct the stale range-partitioning strict floating-point rule (#6049)
Both the shuffle review skill and the native shuffle contributor guide said
that native range partitioning rejects float and double keys when
spark.comet.exec.strictFloatingPoint is enabled. That was true when it was
written but #5981 landed since: supportedRangePartitioningDataType now returns
true for FloatType and DoubleType with no strict-mode condition, because the
native range partitioner normalizes its comparison keys and its sampled
boundary rows the same way the native sort does, and
CometSortOrder.getSupportLevel returns Compatible() to match.
As written the skill told a reviewer to flag a correct native plan as a bug,
and the guide documented a fallback that does not happen.
The rest of the restriction is unchanged: range keys are still primitive-only,
with nested types rejected because native cannot sort them and collated
strings because Comet compares raw bytes. Strict floating point still governs
floating-point values nested in arrays, structs, or maps (#5507), but those
are already rejected as range keys for being nested, so it is not an
additional condition at this level.
---
.ai/skills/review-comet-shuffle-pr/SKILL.md | 17 ++++++++++++-----
docs/source/contributor-guide/native_shuffle.md | 7 +++++--
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/.ai/skills/review-comet-shuffle-pr/SKILL.md b/.ai/skills/review-comet-shuffle-pr/SKILL.md
index 44f00bdaaf..b768923ae7 100644
--- a/.ai/skills/review-comet-shuffle-pr/SKILL.md
+++ b/.ai/skills/review-comet-shuffle-pr/SKILL.md
@@ -52,8 +52,15 @@ keys**, and it is not the same rule for the two partitionings:
- **`RangePartitioning` is primitive-only, unconditionally.**
`supportedRangePartitioningDataType` rejects every nested type, because native cannot sort them,
- and rejects collated strings, because Comet compares raw bytes. It also rejects float and double
- when `spark.comet.exec.strictFloatingPoint` is on.
+ and rejects collated strings, because Comet compares raw bytes. That is the whole rule. Scalar
+ float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since
+ [#5981](https://github.com/apache/datafusion-comet/pull/5981) the native range partitioner
+ normalizes its comparison keys and its sampled boundary rows the same way the native sort does,
+ and `CometSortOrder.getSupportLevel` returns `Compatible()` for them regardless of strict mode.
+ What strict mode still governs is floating point _nested_ in an array, struct, or map
+ ([#5507](https://github.com/apache/datafusion-comet/issues/5507)), which is already out as a
+ range key for being nested. `CometNativeShuffleSuite` runs "range partitioning on floating-point
+ uses native shuffle" under both settings of the config.
- **`HashPartitioning` is primitive-only by default only.** With
`spark.comet.shuffle.native.partitioning.hash.nested.enabled=true` (default `false`),
`supportedHashPartitioningDataType` admits structs and arrays recursively, and maps on Spark
@@ -63,9 +70,9 @@ keys**, and it is not the same rule for the two partitionings:
(`spark/src/main/spark-4.x/`) supports for scalar map keys only.
`CometNativeShuffleSuite` covers both settings of the config.
-A native exchange on a struct or array hash key is therefore not automatically a bug. Check the
-config before calling it one, and do not carry the range-partitioning rule over to hash
-partitioning.
+A native exchange on a struct or array hash key is therefore not automatically a bug, and neither
+is one on a scalar float or double range key in strict mode. Check the config before calling either
+one, and do not carry the range-partitioning rule over to hash partitioning.
- [ ] A PR that widens what native shuffle supports updates the fallback conditions in
`CometShuffleExchangeExec` **and** both docs' "When X is Used" lists
diff --git a/docs/source/contributor-guide/native_shuffle.md b/docs/source/contributor-guide/native_shuffle.md
index 6d7f4a18fa..8d84aa2f6b 100644
--- a/docs/source/contributor-guide/native_shuffle.md
+++ b/docs/source/contributor-guide/native_shuffle.md
@@ -61,8 +61,11 @@ Native shuffle (`CometExchange`) is selected when all of the following condition
columns. Complex types are fully supported as data columns in native shuffle.
- `RangePartitioning` keys must be primitive. `supportedRangePartitioningDataType` rejects every
nested type, because native cannot sort them, and rejects collated strings, because Comet
- compares raw bytes. Float and double are also rejected when
- `spark.comet.exec.strictFloatingPoint` is enabled.
+ compares raw bytes. Scalar float and double are supported, including when
+ `spark.comet.exec.strictFloatingPoint` is enabled, because the native range partitioner
+ normalizes its comparison keys and its sampled boundary rows the same way the native sort
+ does. Strict floating point only affects floating-point values nested in arrays, structs, or
+ maps, which are rejected as range keys for being nested anyway.
- `HashPartitioning` keys must be primitive **by default**. Setting
`spark.comet.shuffle.native.partitioning.hash.nested.enabled` to `true` admits structs and
arrays as keys, checked recursively to their leaves, and maps on Spark 4.0 and later, where
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org