You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/07/28 17:18:06 UTC
(datafusion-comet) branch main updated: docs: fix references to configuration keys that do not exist (#5063)
This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 5fd68c219a docs: fix references to configuration keys that do not exist (#5063)
5fd68c219a is described below
commit 5fd68c219a1df2f7d5b1e12be5c23ea974cfea83
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Jul 28 11:18:01 2026 -0600
docs: fix references to configuration keys that do not exist (#5063)
* refactor: unify shuffle configs under `spark.comet.shuffle.*` prefix
Shuffle-related configs were spread across four disjoint prefixes:
`spark.comet.exec.shuffle.*`, `spark.comet.columnar.shuffle.*`,
`spark.comet.native.shuffle.*`, and `spark.comet.shuffle.*`. This
consolidates all shuffle configs under `spark.comet.shuffle.*` with
`.columnar.` and `.native.` sub-namespaces for mode-specific settings.
Every renamed key is registered via `.withAlternative(...)`, so
existing user configurations continue to work (a deprecation warning
is logged when an old key is read). Per-mode leaves that were
previously misplaced move to their honest namespace:
- `spark.comet.exec.shuffle.writeBufferSize` (native-only) becomes
`spark.comet.shuffle.native.writeBufferSize`.
- `spark.comet.shuffle.preferDictionary.ratio` (JVM-only) becomes
`spark.comet.shuffle.columnar.preferDictionary.ratio`.
The `spark.comet.columnar.shuffle.*` keys also fix Category 2 in
apache/datafusion-comet#4978 (dots-in-segment): `spill.threshold`,
`memory.factor`, and `batch.size` become `spillThreshold`,
`memoryFactor`, and `batchSize`.
Part of apache/datafusion-comet#4978.
* refactor: rename `.columnar.` sub-namespace to `.jvm.`
The JVM-side shuffle path in Comet is not the only columnar shuffle —
native shuffle is also columnar (both use Arrow). The distinguishing
axis is *where* the shuffle logic runs. Renaming the sub-namespace
to `.jvm.` aligns it with the mode value users already set:
`spark.comet.shuffle.mode = jvm`.
Keys renamed:
- spark.comet.shuffle.columnar.spillThreshold → spark.comet.shuffle.jvm.spillThreshold
- spark.comet.shuffle.columnar.memoryFactor → spark.comet.shuffle.jvm.memoryFactor
- spark.comet.shuffle.columnar.batchSize → spark.comet.shuffle.jvm.batchSize
- spark.comet.shuffle.columnar.preferDictionary.ratio → spark.comet.shuffle.jvm.preferDictionary.ratio
The pre-existing deprecated aliases (`spark.comet.columnar.shuffle.*`)
are untouched — they were never `.columnar.`-scoped under the new
prefix. Val names follow: `COMET_SHUFFLE_COLUMNAR_*` → `COMET_SHUFFLE_JVM_*`.
The Scala/Java class names (`CometColumnarShuffle`, `CometColumnarShuffleSuite`, etc.)
still use "columnar" and are left alone — that is a separate rename to
consider once the config vocabulary settles.
* chore: re-run prettier on shuffle docs
Table column alignment tweaks after the `.columnar.` → `.jvm.` key
rename. Content unchanged.
* style: apply spotless formatting to CometShuffleExternalSorter
* refactor: move maxBufferBytes under spark.comet.shuffle.native prefix
* test: update Spark diffs for renamed shuffle configs
Regenerate the Spark SQL test diffs to reference shuffle configs by their
string keys rather than CometConf constants, so the config rename in this
PR no longer breaks test compilation.
- spark.comet.exec.shuffle.enabled -> spark.comet.shuffle.enabled
- CometConf.COMET_EXEC_SHUFFLE_WITH_RANGE_PARTITIONING_ENABLED.key ->
"spark.comet.shuffle.native.partitioning.range.enabled"
* docs: refresh stale issue references and normalize issue link format (#5062)
Audit the user guide and contributor guide for references to issues that
have since been closed, and make the remaining issue and PR links use a
single format.
Thirteen places described a limitation that no longer exists, verified
against the current serde and native code rather than the issue state
alone: size() over MapType, try_mod / EvalMode.TRY, CAST(map AS map),
spark.sql.legacy.castComplexTypesToString, replace with an empty search
string, initcap, str_to_map with the legacy truncate flag, AVG(decimal)
over a window, Percentile, the pyarrow-udf row round-trip, the
datafusion-spark migration epic, the awslabs TPC-DS epic, and the
interval type epic.
Also drop or repoint links that pointed at a closed or unrelated tracker:
the #4098 rows in expressions.md, translate, the TimestampNTZ scan note,
and the from_utc_timestamp timezone-parser note.
Convert bare URLs and bare #NNNN mentions to inline [#NNNN](url), keeping
reference-style definitions where a file already used them, and point URLs
at /pull/ where the number is a pull request.
* docs: fix references to configuration keys that do not exist
Several docs referenced configuration keys that were never registered or were
renamed. Corrected the expression/operator opt-in prefix (spark.comet.expr.* ->
spark.comet.expression.* / spark.comet.operator.*), removed references to a
global allowIncompatible key that does not exist, and updated stale shuffle,
memory pool, and metrics keys.
---
.claude/skills/audit-comet-expression/SKILL.md | 7 +++----
benchmarks/README.md | 3 +--
docs/source/about/versioning_policy.md | 2 +-
docs/source/contributor-guide/adding_a_new_expression.md | 6 +++---
docs/source/contributor-guide/benchmark-results/tpc-ds.md | 2 +-
docs/source/contributor-guide/benchmark-results/tpc-h.md | 2 +-
docs/source/contributor-guide/benchmarking_macos.md | 4 +---
docs/source/contributor-guide/benchmarking_spark_sql_perf.md | 3 +--
docs/source/contributor-guide/expression-audits/generator_funcs.md | 4 ++--
docs/source/contributor-guide/expression-audits/predicate_funcs.md | 2 +-
docs/source/contributor-guide/expression-audits/string_funcs.md | 6 +++---
docs/source/contributor-guide/native_shuffle.md | 2 +-
docs/source/user-guide/latest/metrics.md | 2 +-
13 files changed, 20 insertions(+), 25 deletions(-)
diff --git a/.claude/skills/audit-comet-expression/SKILL.md b/.claude/skills/audit-comet-expression/SKILL.md
index da1c4082d1..e864d23fa8 100644
--- a/.claude/skills/audit-comet-expression/SKILL.md
+++ b/.claude/skills/audit-comet-expression/SKILL.md
@@ -157,10 +157,9 @@ most common bugs in Comet serdes are misalignments between them.
is not implemented, or when running the native path would crash or error.
- `Incompatible(Some(reason))`: Comet can run this, but results may differ
from Spark. The dispatcher only allows it when
- `spark.comet.expr.allowIncompatible=true` (or the per-expression
- equivalent). Use this for known result differences such as locale
- sensitivity, timezone handling, ordering ambiguity, or floating-point
- precision.
+ `spark.comet.expression.<exprName>.allowIncompatible=true` is set. Use
+ this for known result differences such as locale sensitivity, timezone
+ handling, ordering ambiguity, or floating-point precision.
- `Compatible(None)`: full Spark compatibility for this combination of
inputs and options.
- `Compatible(Some(note))`: fully compatible but with a docs-only caveat.
diff --git a/benchmarks/README.md b/benchmarks/README.md
index a35b0e6c99..a95d2e2d3d 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -83,8 +83,7 @@ $SPARK_HOME/bin/spark-submit \
--conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
--conf spark.comet.enabled=true \
--conf spark.comet.exec.enabled=true \
- --conf spark.comet.exec.all.enabled=true \
- --conf spark.comet.cast.allowIncompatible=true \
+ --conf spark.comet.expression.Cast.allowIncompatible=true \
--conf spark.comet.shuffle.enabled=true \
--conf spark.comet.shuffle.mode=auto \
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
diff --git a/docs/source/about/versioning_policy.md b/docs/source/about/versioning_policy.md
index 2d795ff828..ee81593cdc 100644
--- a/docs/source/about/versioning_policy.md
+++ b/docs/source/about/versioning_policy.md
@@ -67,7 +67,7 @@ subsequent releases.
Items whose support level is `Incompatible` or `Unsupported` have no result-compatibility
guarantees. `Incompatible` items require an explicit per-expression or per-operator opt-in
-(for example, `spark.comet.expr.<Name>.allowIncompatible=true`).
+(for example, `spark.comet.expression.<Name>.allowIncompatible=true`).
For details on per-expression and per-operator support levels, see the
[compatibility guide](../user-guide/latest/compatibility/index.md).
diff --git a/docs/source/contributor-guide/adding_a_new_expression.md b/docs/source/contributor-guide/adding_a_new_expression.md
index 06454a19b6..66d2e8b3b1 100644
--- a/docs/source/contributor-guide/adding_a_new_expression.md
+++ b/docs/source/contributor-guide/adding_a_new_expression.md
@@ -145,7 +145,7 @@ A few things to note:
The `SupportLevel` sealed trait has three possible values:
- **`Compatible(notes: Option[String] = None)`** - Comet supports this expression with full compatibility with Spark, or may have known differences in specific edge cases unlikely to affect most users. This is the default if you don't override `getSupportLevel`.
-- **`Incompatible(notes: Option[String] = None)`** - Comet supports this expression but results can differ from Spark. The expression will only be used if `spark.comet.expr.allowIncompatible=true` or the expression-specific config `spark.comet.expr.<exprName>.allowIncompatible=true` is set.
+- **`Incompatible(notes: Option[String] = None)`** - Comet supports this expression but results can differ from Spark. The expression will only be used if the expression-specific config `spark.comet.expression.<exprName>.allowIncompatible=true` is set.
- **`Unsupported(notes: Option[String] = None)`** - Comet does not support this expression under the current conditions. Spark will fall back to its native execution.
All three accept an optional `notes` parameter to provide additional context that is logged for debugging.
@@ -240,11 +240,11 @@ This expression will only be used when users explicitly enable incompatible expr
When the query planner encounters an expression:
-1. It first checks if the expression is explicitly disabled via `spark.comet.expr.<exprName>.enabled=false`
+1. It first checks if the expression is explicitly disabled via `spark.comet.expression.<exprName>.enabled=false`
2. It then calls `getSupportLevel` on the expression handler
3. Based on the result:
- `Compatible()`: Expression proceeds to conversion
- - `Incompatible()`: Expression is skipped unless `spark.comet.expr.allowIncompatible=true` or expression-specific allow config is set
+ - `Incompatible()`: Expression is skipped unless `spark.comet.expression.<exprName>.allowIncompatible=true` is set
- `Unsupported()`: Expression is skipped and a fallback to Spark occurs
Any notes provided will be logged to help with debugging and understanding why an expression was not used.
diff --git a/docs/source/contributor-guide/benchmark-results/tpc-ds.md b/docs/source/contributor-guide/benchmark-results/tpc-ds.md
index 7a988a5080..0e96b977d0 100644
--- a/docs/source/contributor-guide/benchmark-results/tpc-ds.md
+++ b/docs/source/contributor-guide/benchmark-results/tpc-ds.md
@@ -67,5 +67,5 @@ spark.executor.memory=64G
spark.executor.memoryOverhead=10G
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=32G
-spark.comet.memoryPool.fraction=0.8
+spark.comet.exec.memoryPool.fraction=0.8
```
diff --git a/docs/source/contributor-guide/benchmark-results/tpc-h.md b/docs/source/contributor-guide/benchmark-results/tpc-h.md
index eab15fb84a..cb6dc8873f 100644
--- a/docs/source/contributor-guide/benchmark-results/tpc-h.md
+++ b/docs/source/contributor-guide/benchmark-results/tpc-h.md
@@ -75,5 +75,5 @@ spark.memory.offHeap.size=32G
```properties
spark.comet.exec.forceShuffledHashJoin=true
-spark.comet.memoryPool.fraction=0.8
+spark.comet.exec.memoryPool.fraction=0.8
```
diff --git a/docs/source/contributor-guide/benchmarking_macos.md b/docs/source/contributor-guide/benchmarking_macos.md
index 70135ed210..07e4b27105 100644
--- a/docs/source/contributor-guide/benchmarking_macos.md
+++ b/docs/source/contributor-guide/benchmarking_macos.md
@@ -152,10 +152,8 @@ $SPARK_HOME/bin/spark-submit \
--conf spark.plugins=org.apache.spark.CometPlugin \
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
--conf spark.comet.enabled=true \
- --conf spark.comet.exec.shuffle.enableFastEncoding=true \
- --conf spark.comet.exec.shuffle.fallbackToColumnar=true \
+ --conf spark.comet.shuffle.enabled=true \
--conf spark.comet.exec.forceShuffledHashJoin=true \
- --conf spark.comet.expression.allowIncompatible=true \
$DF_BENCH/runners/datafusion-comet/tpcbench.py \
--benchmark tpch \
--data $BENCH_DATA/tpch-data/ \
diff --git a/docs/source/contributor-guide/benchmarking_spark_sql_perf.md b/docs/source/contributor-guide/benchmarking_spark_sql_perf.md
index 4503b414ae..52e380185e 100644
--- a/docs/source/contributor-guide/benchmarking_spark_sql_perf.md
+++ b/docs/source/contributor-guide/benchmarking_spark_sql_perf.md
@@ -240,10 +240,9 @@ $SPARK_HOME/bin/spark-shell \
--conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
--conf spark.comet.enabled=true \
--conf spark.comet.exec.enabled=true \
- --conf spark.comet.exec.all.enabled=true \
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
--conf spark.comet.shuffle.enabled=true \
- --conf spark.comet.columnar.shuffle.enabled=true
+ --conf spark.comet.shuffle.mode=jvm
```
Then register tables and run the benchmark the same way as the Spark baseline, changing the tags and result
diff --git a/docs/source/contributor-guide/expression-audits/generator_funcs.md b/docs/source/contributor-guide/expression-audits/generator_funcs.md
index 3d797e83b3..1b7c2dca03 100644
--- a/docs/source/contributor-guide/expression-audits/generator_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/generator_funcs.md
@@ -27,7 +27,7 @@
## explode_outer
-- Same `CometExplodeExec` path as `explode`, but the `outer=true` case is `Incompatible` (empty arrays are not preserved as null outputs) and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
+- Same `CometExplodeExec` path as `explode`, but the `outer=true` case is `Incompatible` (empty arrays are not preserved as null outputs) and falls back unless `spark.comet.operator.GenerateExec.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
## posexplode
@@ -35,6 +35,6 @@
## posexplode_outer
-- Same `CometExplodeExec` path as `posexplode`, but the `outer=true` case is `Incompatible` and falls back unless `spark.comet.expr.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
+- Same `CometExplodeExec` path as `posexplode`, but the `outer=true` case is `Incompatible` and falls back unless `spark.comet.operator.GenerateExec.allowIncompatible=true` ([datafusion#19053](https://github.com/apache/datafusion/issues/19053)).
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/docs/source/contributor-guide/expression-audits/predicate_funcs.md b/docs/source/contributor-guide/expression-audits/predicate_funcs.md
index 74d32b67ab..57a55262ca 100644
--- a/docs/source/contributor-guide/expression-audits/predicate_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/predicate_funcs.md
@@ -146,6 +146,6 @@
## rlike
-- See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR [#4461](https://github.com/apache/datafusion-comet/pull/4461)). Uses the Rust `regex` crate, which differs from Java's `Pattern` engine; requires `spark.comet.expression.regexp.allowIncompatible=true`.
+- See `string_funcs / regexp_replace` and the `CometRLike` notes (audited in PR [#4461](https://github.com/apache/datafusion-comet/pull/4461)). By default `CometRLike` routes through the JVM codegen dispatcher so Spark's own `Pattern` engine runs inside the Comet pipeline. The native path uses the Rust `regex` crate, which differs from Java's `Pattern` engine, so it is opt-in via `spark.comet.expression.RLike.allowIncompatible=true` and only applies when the pattern is a string literal.
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/docs/source/contributor-guide/expression-audits/string_funcs.md b/docs/source/contributor-guide/expression-audits/string_funcs.md
index f0bcb18ce2..58b300ebc1 100644
--- a/docs/source/contributor-guide/expression-audits/string_funcs.md
+++ b/docs/source/contributor-guide/expression-audits/string_funcs.md
@@ -129,7 +129,7 @@
## lower
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale `toLowerCase` on `UTF8String`. Comet routes to DataFusion `lower` (Rust Unicode default case mapping, no locale awareness) and is unconditionally `Incompatible`; users opt in via the standard `spark.comet.expression.Lower.allowIncompatible=true`.
+- Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale `toLowerCase` on `UTF8String`. Comet routes through the JVM codegen dispatcher by default, so results are byte-exact. The native path (DataFusion `lower`, Rust Unicode default case mapping with no locale awareness) is opt-in via `spark.comet.caseConversion.enabled=true`.
- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Lower.exec(v, collationId, useICU)` with `SQLConf.ICU_CASE_MAPPINGS_ENABLED`; `inputTypes` widened to `StringTypeWithCollation`. Comet ignores collation and ICU mode, so non-default collations or `ICU_CASE_MAPPINGS_ENABLED=true` diverge even after opting in ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
## lpad
@@ -157,7 +157,7 @@
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
- Spark 3.5.8 (audited 2026-05-27): baseline. `RegExpReplace(subject, regexp, rep, pos)` with foldable `pos > 0`; uses Java `Pattern`. Comet supports only `pos = 1` (other offsets fall back) and injects a `'g'` flag because DataFusion's `regexp_replace` stops at the first match by default.
- Spark 4.0.1 (audited 2026-05-27): adds raw-string literal support at the parser level and `nullIntolerant: Boolean = true`; runtime semantics unchanged.
-- Known limitation: regex semantics differ (Rust `regex` crate vs Java `Pattern`); `RegExp.isSupportedPattern` currently returns `false` for every pattern, so the path always requires `spark.comet.expression.regexp.allowIncompatible=true`.
+- Known limitation: regex semantics differ (Rust `regex` crate vs Java `Pattern`). `CometRegExpReplace` therefore routes through the JVM codegen dispatcher by default; the native path is opt-in via `spark.comet.expression.RegExpReplace.allowIncompatible=true` and only applies when `pos` is the literal `1`.
## repeat
@@ -248,7 +248,7 @@
## upper
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
-- Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale `toUpperCase` on `UTF8String`. Comet routes to DataFusion `upper` (Rust Unicode default case mapping, no locale awareness) and is unconditionally `Incompatible`; users opt in via the standard `spark.comet.expression.Upper.allowIncompatible=true`.
+- Spark 3.5.8 (audited 2026-05-27): baseline. JVM default-locale `toUpperCase` on `UTF8String`. Comet routes through the JVM codegen dispatcher by default, so results are byte-exact. The native path (DataFusion `upper`, Rust Unicode default case mapping with no locale awareness) is opt-in via `spark.comet.caseConversion.enabled=true`.
- Spark 4.0.1 (audited 2026-05-27): routes through `CollationSupport.Upper.exec(v, collationId, useICU)` with `SQLConf.ICU_CASE_MAPPINGS_ENABLED`. Comet does not propagate collation or ICU mode; non-default collations or `ICU_CASE_MAPPINGS_ENABLED=true` diverge even after opting in ([#2190](https://github.com/apache/datafusion-comet/issues/2190)).
[Spark Expression Support]: ../../user-guide/latest/expressions.md
diff --git a/docs/source/contributor-guide/native_shuffle.md b/docs/source/contributor-guide/native_shuffle.md
index 9cd9ebc6c5..e8cfe81ec3 100644
--- a/docs/source/contributor-guide/native_shuffle.md
+++ b/docs/source/contributor-guide/native_shuffle.md
@@ -256,7 +256,7 @@ independently compressed, allowing parallel decompression during reads.
| `spark.comet.shuffle.mode` | `auto` | Shuffle mode: `native`, `jvm`, or `auto` |
| `spark.comet.shuffle.compression.codec` | `zstd` | Compression codec |
| `spark.comet.shuffle.compression.zstd.level` | `1` | Zstd compression level |
-| `spark.comet.shuffle.write.buffer.size` | `1MB` | Write buffer size |
+| `spark.comet.shuffle.native.writeBufferSize` | `1MB` | Write buffer size |
| `spark.comet.shuffle.jvm.batchSize` | `8192` | Target rows per batch |
## Comparison with JVM Shuffle
diff --git a/docs/source/user-guide/latest/metrics.md b/docs/source/user-guide/latest/metrics.md
index 0579b30818..2e36aba291 100644
--- a/docs/source/user-guide/latest/metrics.md
+++ b/docs/source/user-guide/latest/metrics.md
@@ -21,7 +21,7 @@ under the License.
## Spark SQL Metrics
-Set `spark.comet.metrics.detailed=true` to see all available Comet metrics.
+Comet operators report the following metrics in the Spark SQL UI.
### CometScanExec
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org