(datafusion-comet) branch main updated: docs: explain native vs codegen-dispatch implementation model and allowIncompatible opt-in (#4629)

[ag...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/main by this push:
     new 754289ac7 docs: explain native vs codegen-dispatch implementation model and allowIncompatible opt-in (#4629)
754289ac7 is described below

commit 754289ac7d693bb97dc29d22664a6334d8bfdaff
Author: Andy Grove <ag...@apache.org>
AuthorDate: Fri Jun 12 11:43:38 2026 -0600

    docs: explain native vs codegen-dispatch implementation model and allowIncompatible opt-in (#4629)
---
 .../user-guide/latest/compatibility/index.md       | 27 ++++++++++++++++++++++
 docs/source/user-guide/latest/expressions.md       | 10 +++++---
 2 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md
index 46578d168..54a7b4ae3 100644
--- a/docs/source/user-guide/latest/compatibility/index.md
+++ b/docs/source/user-guide/latest/compatibility/index.md
@@ -30,3 +30,30 @@ This guide documents areas where Comet's behavior is known to differ from Spark.
 - **Expressions**: per-expression compatibility notes, including cast.
 - **JSON**: choosing between the native and Spark-compatible engines for JSON expressions.
 - **Spark versions**: version-specific known issues and limitations.
+
+## Native and codegen-dispatch implementations
+
+Some Spark expressions have two implementations in Comet:
+
+- A **codegen-dispatch** implementation that runs Spark's own generated code for the
+  expression inside Comet's native pipeline (via the Arrow-direct codegen dispatcher). This
+  produces byte-exact Spark results at the cost of one JNI round-trip per batch. It is gated
+  globally by `spark.comet.exec.scalaUDF.codegen.enabled` (enabled by default); when the
+  dispatcher is disabled, these expressions fall back to Spark.
+- A **native** (Rust / DataFusion) implementation that is faster, with no JNI overhead, but
+  has known semantic differences from Spark for some inputs or patterns.
+
+Because the codegen-dispatch path matches Spark exactly, Comet uses it by **default**. The
+faster native path is **opt-in per expression** via that expression's
+`spark.comet.expression.<ExprClassName>.allowIncompatible=true` flag, which declares that you
+accept its differences from Spark. There is no global opt-in. When the native path is enabled
+but a specific input or pattern has no native implementation, Comet routes that case back
+through the codegen dispatcher rather than running something incompatible.
+
+This is the model behind the [regular expression](regex.md) and [JSON](json.md) families,
+which document their per-expression configs and the specific differences to expect.
+
+This is distinct from expressions that have **no** codegen-dispatch path: there, the
+incompatible cases fall back to Spark by default, and `allowIncompatible=true` runs the native
+(incompatible) path instead. `cast` is the main example; see the
+[expression reference](../expressions.md) for which expressions have incompatible cases.
diff --git a/docs/source/user-guide/latest/expressions.md b/docs/source/user-guide/latest/expressions.md
index 899fe4b72..66ee762b2 100644
--- a/docs/source/user-guide/latest/expressions.md
+++ b/docs/source/user-guide/latest/expressions.md
@@ -26,10 +26,14 @@ transparently falls back to Spark for that part of the plan; results are unaffec
 
 Expressions marked ✅ Supported are enabled by default and produce Spark-compatible results.
 
-Some ✅ Supported expressions have specific incompatible cases that fall back to Spark by
-default. Those cases must be opted into per expression with
+Some ✅ Supported expressions have specific incompatible cases that are not run by default.
+Those cases must be opted into per expression with
 `spark.comet.expression.EXPRNAME.allowIncompatible=true` (where `EXPRNAME` is the Spark
-expression class name, for example `Cast`). There is no global opt-in.
+expression class name, for example `Cast`). There is no global opt-in. By default such a case
+either falls back to Spark (for example `cast`) or, when the expression has a Spark-compatible
+codegen-dispatch implementation, runs through that instead (for example the regex and JSON
+families). See [Native and codegen-dispatch implementations](compatibility/index.md#native-and-codegen-dispatch-implementations)
+for how Comet chooses.
 
 Most expressions can also be disabled with `spark.comet.expression.EXPRNAME.enabled=false`, where
 `EXPRNAME` is the Spark expression class name (for example `Length` or `StartsWith`). See the


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org