You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by ag...@apache.org on 2026/07/07 19:10:24 UTC
(datafusion-comet) branch main updated: feat: route Unsupported through codegen dispatch for opt-in serdes (#4728)
This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 5076a5351a feat: route Unsupported through codegen dispatch for opt-in serdes (#4728)
5076a5351a is described below
commit 5076a5351a686c2ae86fe59929a5a6058001542c
Author: Andy Grove <ag...@apache.org>
AuthorDate: Tue Jul 7 13:10:19 2026 -0600
feat: route Unsupported through codegen dispatch for opt-in serdes (#4728)
* feat: route Unsupported through codegen dispatch for opt-in serdes
CodegenDispatchFallback already routes Incompatible cases through the
JVM codegen dispatcher so the projection stays inside the Comet pipeline
instead of falling back to Spark. Apply the same routing to Unsupported
cases: when getSupportLevel returns Unsupported and the serde mixes in
CodegenDispatchFallback, run Spark's own doGenCode via the dispatcher
before resorting to Spark fallback. Affects Concat (non-string children),
SortArray (nested Struct/Null children), ArrayIntersect (collated
strings), TruncDate and TruncTimestamp (formats outside the native set).
Also refresh the expression compatibility docs: drop "faster native"
wording (no measured claim), clarify that the default path runs in the
JVM via Spark codegen, and render Unsupported reasons differently for
CodegenDispatchFallback serdes (always JVM dispatch) vs everything else
(always Spark fallback).
* refactor: extract dispatchIfFallback helper for shared dispatch routing
The Unsupported and Incompatible arms both pattern-match on
CodegenDispatchFallback and call emitJvmCodegenDispatch. Lift that
pattern into a single helper that returns the matched handler alongside
the dispatched expression, so the Incompatible arm can reach
nativeOptInConfigKeyOverride for its [COMET-INFO] hint without
re-matching the same value.
* test: update stale fallback expectations for serdes now routed through codegen dispatch
Concat, SortArray, TruncDate, and TruncTimestamp now route their previously Unsupported cases through the JVM codegen dispatcher and stay native instead of falling back to Spark. Update the test expectations to assert native execution and a matching answer rather than a fallback reason.
* test: cover SortArray non-literal ascendingOrder and dispatcher-disabled fallback
Add a SQL test exercising the non-boolean-literal ascendingOrder branch of
SortArray, which routes through codegen dispatch and stays native, and a
dedicated test with spark.comet.exec.scalaUDF.codegen.enabled=false confirming
the Unsupported CodegenDispatchFallback cases (Concat, SortArray, TruncDate)
fall back to Spark cleanly. Log at debug level when an Unsupported case routes
through the dispatcher.
* test: gate non-literal sort_array ascendingOrder cases to Spark 4.0+
Spark 3.4/3.5's SortArray requires the ascendingOrder argument to be a boolean
literal and rejects a non-literal expression at analysis time, so the
sort_array(arr, true AND true) / (false OR false) cases fail on those versions.
Move them into sort_array_min_spark_4_0.sql gated with MinSparkVersion 4.0,
where Spark accepts foldable non-literal sort orders.
---
.../latest/compatibility/expressions/index.md | 2 +-
.../compatibility/expressions/spark-3.4/index.md | 2 +-
.../compatibility/expressions/spark-3.5/index.md | 2 +-
.../compatibility/expressions/spark-4.0/index.md | 2 +-
.../compatibility/expressions/spark-4.1/index.md | 2 +-
.../user-guide/latest/compatibility/index.md | 8 +--
.../main/scala/org/apache/comet/GenerateDocs.scala | 32 ++++++++---
.../apache/comet/serde/CometExpressionSerde.scala | 30 +++++++---
.../org/apache/comet/serde/QueryPlanSerde.scala | 50 ++++++++++++----
.../main/scala/org/apache/comet/serde/arrays.scala | 3 +-
.../sql-tests/expressions/array/array_concat.sql | 13 +++--
.../sql-tests/expressions/array/sort_array.sql | 15 ++++-
.../expressions/array/sort_array_min_spark_4_0.sql | 44 ++++++++++++++
.../misc/codegen_dispatch_disabled_fallback.sql | 67 ++++++++++++++++++++++
.../sql-tests/expressions/string/concat.sql | 13 +++--
.../comet/CometTemporalExpressionSuite.scala | 28 +++++----
16 files changed, 248 insertions(+), 65 deletions(-)
diff --git a/docs/source/user-guide/latest/compatibility/expressions/index.md b/docs/source/user-guide/latest/compatibility/expressions/index.md
index 3df1d9f4c0..3574699195 100644
--- a/docs/source/user-guide/latest/compatibility/expressions/index.md
+++ b/docs/source/user-guide/latest/compatibility/expressions/index.md
@@ -35,7 +35,7 @@ Spark 4.1 <spark-4.1/index>
Expressions that are not 100% Spark-compatible fall back to Spark by default, except those
with a JVM codegen-dispatch path, which stay in Comet's native pipeline and match Spark
exactly. Set `spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is
-the Spark expression class name, to run Comet's faster native implementation despite its
+the Spark expression class name, to run Comet's native implementation despite its
differences from Spark. See the
[Comet Supported Expressions Guide](../../expressions.md) for more information on this
configuration setting.
diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md
index 8dc1b26c93..40fe54c4ea 100644
--- a/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md
+++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.4/index.md
@@ -23,7 +23,7 @@ Compatibility notes for Comet running on Apache Spark 3.4. Expressions that are
Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch
path, which stay in Comet's native pipeline and match Spark exactly. Set
`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark
-expression class name, to run Comet's faster native implementation despite its differences
+expression class name, to run Comet's native implementation despite its differences
from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md)
for more information on this configuration setting.
diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md
index 57536d768c..db11f25083 100644
--- a/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md
+++ b/docs/source/user-guide/latest/compatibility/expressions/spark-3.5/index.md
@@ -23,7 +23,7 @@ Compatibility notes for Comet running on Apache Spark 3.5. Expressions that are
Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch
path, which stay in Comet's native pipeline and match Spark exactly. Set
`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark
-expression class name, to run Comet's faster native implementation despite its differences
+expression class name, to run Comet's native implementation despite its differences
from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md)
for more information on this configuration setting.
diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md
index 659b830e9c..371da9a49f 100644
--- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md
+++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.0/index.md
@@ -23,7 +23,7 @@ Compatibility notes for Comet running on Apache Spark 4.0. Expressions that are
Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch
path, which stay in Comet's native pipeline and match Spark exactly. Set
`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark
-expression class name, to run Comet's faster native implementation despite its differences
+expression class name, to run Comet's native implementation despite its differences
from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md)
for more information on this configuration setting.
diff --git a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md
index aa2f4a130b..aaca6b485b 100644
--- a/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md
+++ b/docs/source/user-guide/latest/compatibility/expressions/spark-4.1/index.md
@@ -23,7 +23,7 @@ Compatibility notes for Comet running on Apache Spark 4.1. Expressions that are
Spark-compatible fall back to Spark by default, except those with a JVM codegen-dispatch
path, which stay in Comet's native pipeline and match Spark exactly. Set
`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark
-expression class name, to run Comet's faster native implementation despite its differences
+expression class name, to run Comet's native implementation despite its differences
from Spark. See the [Comet Supported Expressions Guide](../../../expressions.md)
for more information on this configuration setting.
diff --git a/docs/source/user-guide/latest/compatibility/index.md b/docs/source/user-guide/latest/compatibility/index.md
index f3b531a18f..52a5ed6850 100644
--- a/docs/source/user-guide/latest/compatibility/index.md
+++ b/docs/source/user-guide/latest/compatibility/index.md
@@ -34,7 +34,7 @@ This guide documents areas where Comet's behavior is known to differ from Spark.
## Compatible by default, opt in to native
Comet runs a Spark-compatible implementation of every supported expression by default. Some
-expressions also have a faster native implementation that can differ from Spark for certain
+expressions also have a native implementation that can differ from Spark for certain
inputs. These are not used unless you opt in by setting the relevant
`spark.comet.expression.<Name>.allowIncompatible=true` config (a few use a dedicated config, noted
per expression below), after which you accept the documented differences.
@@ -53,11 +53,11 @@ Some Spark expressions have two implementations in Comet:
produces byte-exact Spark results at the cost of one JNI round-trip per batch. It is gated
globally by `spark.comet.exec.scalaUDF.codegen.enabled` (enabled by default); when the
dispatcher is disabled, these expressions fall back to Spark.
-- A **native** (Rust / DataFusion) implementation that is faster, with no JNI overhead, but
- has known semantic differences from Spark for some inputs or patterns.
+- A **native** (Rust / DataFusion) implementation that avoids the JNI round-trip but has
+ known semantic differences from Spark for some inputs or patterns.
Because the codegen-dispatch path matches Spark exactly, Comet uses it by **default**. The
-faster native path is **opt-in per expression** via that expression's
+native path is **opt-in per expression** via that expression's
`spark.comet.expression.<ExprClassName>.allowIncompatible=true` flag, which declares that you
accept its differences from Spark. There is no global opt-in. When the native path is enabled
but a specific input or pattern has no native implementation, Comet routes that case back
diff --git a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
index 195e6a05bf..9ea512c40c 100644
--- a/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
+++ b/spark/src/main/scala/org/apache/comet/GenerateDocs.scala
@@ -28,7 +28,7 @@ import org.apache.spark.sql.catalyst.expressions.Cast
import org.apache.comet.CometConf.COMET_ONHEAP_MEMORY_OVERHEAD
import org.apache.comet.expressions.{CometCast, CometEvalMode}
-import org.apache.comet.serde.{CometAggregateExpressionSerde, CometExpressionSerde, Compatible, Incompatible, NativeOptInAvailable, QueryPlanSerde, Unsupported}
+import org.apache.comet.serde.{CodegenDispatchFallback, CometAggregateExpressionSerde, CometExpressionSerde, Compatible, Incompatible, NativeOptInAvailable, QueryPlanSerde, Unsupported}
/**
* Utility for generating markdown documentation from the configs.
@@ -49,12 +49,15 @@ object GenerateDocs {
* @param incompatibleReasons
* reasons the native implementation is incompatible with Spark
* @param unsupportedReasons
- * cases that Comet does not support
+ * cases that Comet's native implementation does not handle
* @param nativeOptIn
* whether the serde implements `NativeOptInAvailable`, meaning the expression runs a
- * Spark-compatible path by default and the user can opt into a faster native path
+ * Spark-compatible path by default and the user can opt into a native path
* @param nativeOptInConfigKey
* the config key the user sets to opt into the native path
+ * @param codegenDispatchFallback
+ * whether the serde mixes in `CodegenDispatchFallback`, meaning `unsupportedReasons` cases
+ * route through the JVM codegen dispatcher instead of falling back to Spark
*/
private case class ExprNotes(
name: String,
@@ -62,7 +65,8 @@ object GenerateDocs {
incompatibleReasons: Seq[String],
unsupportedReasons: Seq[String],
nativeOptIn: Boolean,
- nativeOptInConfigKey: String)
+ nativeOptInConfigKey: String,
+ codegenDispatchFallback: Boolean)
private type CategoryNotes = Seq[ExprNotes]
@@ -80,7 +84,8 @@ object GenerateDocs {
serde.getIncompatibleReasons(),
serde.getUnsupportedReasons(),
optIn,
- key)
+ key,
+ codegenDispatchFallback = serde.isInstanceOf[CodegenDispatchFallback])
}
/** Build the documentation notes for a single aggregate expression serde. */
@@ -92,7 +97,8 @@ object GenerateDocs {
serde.getUnsupportedReasons(),
// Aggregate serdes do not have a native opt-in path.
nativeOptIn = false,
- nativeOptInConfigKey = CometConf.getExprAllowIncompatConfigKey(cls))
+ nativeOptInConfigKey = CometConf.getExprAllowIncompatConfigKey(cls),
+ codegenDispatchFallback = false)
/**
* Mapping from expression category to the compatibility guide filename where that category's
@@ -277,8 +283,9 @@ object GenerateDocs {
}
if (n.incompatibleReasons.nonEmpty) {
val header = if (n.nativeOptIn) {
- s"\nBy default, Comet runs a Spark-compatible implementation of `$name`. Set" +
- s" `${n.nativeOptInConfigKey}=true` to use Comet's faster native implementation" +
+ s"\nBy default, `$name` is evaluated in the JVM using Spark's own code-generated" +
+ " implementation (run inside the Comet pipeline), which matches Spark exactly." +
+ s" Set `${n.nativeOptInConfigKey}=true` to opt into Comet's native implementation" +
" instead, which has the following differences from Spark:\n\n"
} else {
s"\nThe following incompatibilities cause `$name` to fall back to Spark by default." +
@@ -291,7 +298,14 @@ object GenerateDocs {
}
}
if (n.unsupportedReasons.nonEmpty) {
- w.write("\nThe following cases are not supported by Comet:\n\n".getBytes)
+ val header = if (n.codegenDispatchFallback) {
+ "\nThe following cases have no native implementation and always run in the JVM using" +
+ " Spark's code-generated implementation (inside the Comet pipeline):\n\n"
+ } else {
+ "\nThe following cases are not supported by Comet and always fall back to Spark," +
+ " regardless of any `allowIncompatible` setting:\n\n"
+ }
+ w.write(header.getBytes)
for (reason <- n.unsupportedReasons) {
w.write(s"- $reason\n".getBytes)
}
diff --git a/spark/src/main/scala/org/apache/comet/serde/CometExpressionSerde.scala b/spark/src/main/scala/org/apache/comet/serde/CometExpressionSerde.scala
index 5baf2d9bc0..cd810e81f4 100644
--- a/spark/src/main/scala/org/apache/comet/serde/CometExpressionSerde.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/CometExpressionSerde.scala
@@ -63,9 +63,13 @@ trait CometExpressionSerde[T <: Expression] {
def getIncompatibleReasons(): Seq[String] = Seq.empty
/**
- * Get documentation for usages where this expression is unsupported with Spark. This is called
- * from GenerateDocs when generating the Compatibility Guide. Each reason should be written in
- * Markdown and may span multiple lines.
+ * Get documentation for usages of this expression that Comet's native implementation does not
+ * support. Cases listed here normally fall back to Spark, regardless of any `allowIncompatible`
+ * setting. When the serde mixes in `CodegenDispatchFallback` they are instead routed through
+ * the JVM codegen dispatcher (Spark's own `doGenCode` inside the Comet pipeline), so they stay
+ * in the Comet pipeline while still matching Spark exactly. This is called from GenerateDocs
+ * when generating the Compatibility Guide. Each reason should be written in Markdown and may
+ * span multiple lines.
*
* @return
* List of reasons, defaulting to an empty list.
@@ -101,14 +105,22 @@ trait CometExpressionSerde[T <: Expression] {
}
/**
- * Opt-in marker for expression serdes that have a native implementation which is `Incompatible`
- * with Spark for some inputs. When such an expression reports `Incompatible` and the user has not
- * enabled `allowIncompatible` for it, mixing in this trait routes it through the JVM codegen
- * dispatcher (running Spark's own `doGenCode` inside the Comet pipeline) instead of falling the
- * projection back to Spark, so it stays native while still matching Spark exactly.
+ * Mixin for serdes whose native implementation cannot match Spark for some inputs. When
+ * `getSupportLevel` returns `Incompatible` and the user has not enabled `allowIncompatible`, the
+ * expression routes through the JVM codegen dispatcher (Spark's own `doGenCode` inside the Comet
+ * pipeline) instead of falling the projection back to Spark. When `getSupportLevel` returns
+ * `Unsupported`, the expression always routes through the dispatcher -- the serde is declaring
+ * "no native path exists for this case; run Spark's code in-pipeline." Spark fallback is reserved
+ * for the case where the dispatcher itself cannot handle the expression (e.g. the global codegen
+ * flag is off, or the kernel rejects the bound tree).
+ *
+ * Contract for `Unsupported` reasons on a `CodegenDispatchFallback` serde: the case must be
+ * something `Expression.doGenCode` can compile. If you mark something `Unsupported` because Spark
+ * also rejects it, that is fine -- the dispatcher will surface the same error Spark would have.
*
* Enrollment is opt-in: only serdes that explicitly mix this in are routed through the
- * dispatcher. Every other `Incompatible` expression falls back to Spark.
+ * dispatcher. Every other `Incompatible` expression falls back to Spark, and every other
+ * `Unsupported` expression falls back to Spark.
*/
trait CodegenDispatchFallback extends NativeOptInAvailable { self: CometExpressionSerde[_] => }
diff --git a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
index 7146eaec9b..939d1480ff 100644
--- a/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
@@ -781,8 +781,22 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
}
handler.getSupportLevel(expr) match {
case Unsupported(notes) =>
- withFallbackReason(expr, notes.getOrElse(""))
- None
+ // `CodegenDispatchFallback` serdes have no native path for these cases either, but the
+ // dispatcher can still run Spark's own `doGenCode` inside the Comet pipeline. Try that
+ // before falling the projection back to Spark. No `[COMET-INFO]` hint here: unlike
+ // `Incompatible`, there is no native opt-in for the user to flip, so log at debug level
+ // to give anyone tracing why a projection stayed native something to go on.
+ dispatchIfFallback(handler, expr, inputs, binding)
+ .map { case (_, proto) =>
+ logDebug(
+ s"$expr has no native path (${notes.getOrElse("unsupported")}) but is routed " +
+ "through the JVM codegen dispatcher to stay in the Comet pipeline.")
+ proto
+ }
+ .orElse {
+ withFallbackReason(expr, notes.getOrElse(""))
+ None
+ }
case Incompatible(notes) =>
val exprAllowIncompat = CometConf.isExprAllowIncompat(exprConfName)
if (exprAllowIncompat) {
@@ -799,15 +813,12 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
// pipeline) so the projection stays native while still matching Spark. Everything else
// falls back to Spark. Falling back is also the result when the dispatcher cannot
// handle the expression.
- val dispatched = handler match {
- case h: CodegenDispatchFallback =>
- CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding).map { proto =>
- val key = h.nativeOptInConfigKeyOverride
- .getOrElse(CometConf.getExprAllowIncompatConfigKey(exprConfName))
- withInfo(expr, NativeOptIn.message(exprConfName, key))
- proto
- }
- case _ => None
+ val dispatched = dispatchIfFallback(handler, expr, inputs, binding).map {
+ case (h, proto) =>
+ val key = h.nativeOptInConfigKeyOverride
+ .getOrElse(CometConf.getExprAllowIncompatConfigKey(exprConfName))
+ withInfo(expr, NativeOptIn.message(exprConfName, key))
+ proto
}
dispatched.orElse {
val optionalNotes = notes.map(str => s" ($str)").getOrElse("")
@@ -1050,6 +1061,23 @@ object QueryPlanSerde extends Logging with CometExprShim with CometTypeShim {
}
+ /**
+ * If `handler` is a `CodegenDispatchFallback`, run `expr` through the JVM codegen dispatcher
+ * and return `Some((handler, proto))` on success; otherwise return `None`. Shared by the
+ * `Unsupported` and (non-opt-in) `Incompatible` arms of `exprToProtoInternal` so they don't
+ * each inline the same pattern match. Returning the matched handler lets the `Incompatible` arm
+ * reach `nativeOptInConfigKeyOverride` without re-pattern-matching the same value.
+ */
+ private def dispatchIfFallback(
+ handler: CometExpressionSerde[_],
+ expr: Expression,
+ inputs: Seq[Attribute],
+ binding: Boolean): Option[(CodegenDispatchFallback, Expr)] = handler match {
+ case h: CodegenDispatchFallback =>
+ CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding).map(h -> _)
+ case _ => None
+ }
+
// scalastyle:off
/**
* Align w/ Arrow's
diff --git a/spark/src/main/scala/org/apache/comet/serde/arrays.scala b/spark/src/main/scala/org/apache/comet/serde/arrays.scala
index 8eda097ce6..1b39f5cd1f 100644
--- a/spark/src/main/scala/org/apache/comet/serde/arrays.scala
+++ b/spark/src/main/scala/org/apache/comet/serde/arrays.scala
@@ -121,8 +121,7 @@ object CometSortArray extends CometExpressionSerde[SortArray] with CodegenDispat
" floating-point types is not 100% compatible with Spark")
override def getUnsupportedReasons(): Seq[String] = Seq(
- "Nested arrays with `Struct` or `Null` child values are not supported natively and will" +
- " fall back to Spark.")
+ "Nested arrays with `Struct` or `Null` child values are not supported natively")
private def supportedSortArrayElementType(
dt: DataType,
diff --git a/spark/src/test/resources/sql-tests/expressions/array/array_concat.sql b/spark/src/test/resources/sql-tests/expressions/array/array_concat.sql
index 21d83a7afb..95fc3bcbad 100644
--- a/spark/src/test/resources/sql-tests/expressions/array/array_concat.sql
+++ b/spark/src/test/resources/sql-tests/expressions/array/array_concat.sql
@@ -24,17 +24,20 @@ CREATE TABLE test_array_concat(c1 array<int>, c2 array<int>, c3 array<int>, c4 a
statement
INSERT INTO test_array_concat VALUES (array(0, 1), array(2, 3), array(), array(null), null), (array(1, 2), array(3, 4), array(), array(null), null), (array(2, 3), array(4, 5), array(), array(null), null)
-query expect_fallback(CONCAT supports only string input parameters)
+-- Concat mixes in CodegenDispatchFallback, so non-string (array) children have no native path
+-- but route through the JVM codegen dispatcher (Spark's own Concat.doGenCode inside the Comet
+-- pipeline) and stay native while matching Spark exactly.
+query
SELECT concat(c1, c2) AS x FROM test_array_concat
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(c1, c1) AS x FROM test_array_concat
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(c1, c2, c3) AS x FROM test_array_concat
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(c1, c2, c3, c5) AS x FROM test_array_concat
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(concat(c1, c2, c3), concat(c1, c3)) AS x FROM test_array_concat
diff --git a/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql b/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql
index 1ced53394d..dc9e7ec85b 100644
--- a/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql
+++ b/spark/src/test/resources/sql-tests/expressions/array/sort_array.sql
@@ -295,12 +295,18 @@ INSERT INTO test_sort_array_nested_struct VALUES
(array()),
(NULL)
-query expect_fallback(Sort on array element type ArrayType(StructType(StructField(a,IntegerType)
+-- SortArray mixes in CodegenDispatchFallback, so nested arrays with Struct children have no
+-- native path but route through the JVM codegen dispatcher (Spark's own SortArray.doGenCode
+-- inside the Comet pipeline) and stay native while matching Spark exactly.
+query
SELECT sort_array(arr) FROM test_sort_array_nested_struct
-query expect_fallback(Sort on array element type ArrayType(StructType(StructField(a,IntegerType)
+query
SELECT sort_array(arr, false) FROM test_sort_array_nested_struct
+-- Non-literal (foldable) ascendingOrder is exercised in sort_array_min_spark_4_0.sql: Spark 3.4/3.5
+-- reject a non-literal sort order at analysis time, so those cases are gated to Spark 4.0+.
+
-- literal arguments
query
SELECT
@@ -391,7 +397,10 @@ SELECT
sort_array(array(NULL, NULL)),
sort_array(cast(NULL as array<int>))
-query expect_fallback(Sort on array element type ArrayType(StructType(StructField(a,IntegerType)
+-- nested arrays with Struct children have no native path but route through the JVM codegen
+-- dispatcher (Spark's own SortArray.doGenCode inside the Comet pipeline) and stay native while
+-- matching Spark exactly.
+query
SELECT sort_array(
array(
array(named_struct('a', 2)),
diff --git a/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql b/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql
new file mode 100644
index 0000000000..e3e0f6342c
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/array/sort_array_min_spark_4_0.sql
@@ -0,0 +1,44 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ConfigMatrix: parquet.enable.dictionary=false,true
+
+-- Spark 3.4/3.5's SortArray requires the ascendingOrder argument to be a boolean literal and
+-- rejects any other expression at analysis time. Spark 4.0 relaxed this to accept any foldable
+-- boolean, so the non-literal-ascendingOrder cases below can only be exercised on Spark 4.0+.
+-- MinSparkVersion: 4.0
+
+statement
+CREATE TABLE test_sort_array_int(arr array<int>) USING parquet
+
+statement
+INSERT INTO test_sort_array_int VALUES
+ (array(3, 1, 4, 1, 5)),
+ (array(3, NULL, 1, NULL, 2)),
+ (array()),
+ (NULL)
+
+-- ascendingOrder given as a foldable non-literal expression. Spark requires ascendingOrder to be
+-- foldable, but this suite disables constant folding, so the serde sees `And`/`Or` rather than a
+-- boolean literal. That has no native path, but SortArray mixes in CodegenDispatchFallback, so it
+-- routes through the JVM codegen dispatcher (Spark's own SortArray.doGenCode inside the Comet
+-- pipeline) and stays native while matching Spark exactly.
+query
+SELECT sort_array(arr, true AND true) FROM test_sort_array_int
+
+query
+SELECT sort_array(arr, false OR false) FROM test_sort_array_int
diff --git a/spark/src/test/resources/sql-tests/expressions/misc/codegen_dispatch_disabled_fallback.sql b/spark/src/test/resources/sql-tests/expressions/misc/codegen_dispatch_disabled_fallback.sql
new file mode 100644
index 0000000000..eebf40bff1
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/misc/codegen_dispatch_disabled_fallback.sql
@@ -0,0 +1,67 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- Safety net for the CodegenDispatchFallback routing. `Unsupported` cases on serdes that mix in
+-- CodegenDispatchFallback normally stay native by routing through the JVM codegen dispatcher.
+-- With the dispatcher disabled it has no path to take, so these same cases must fall back to
+-- Spark cleanly (matching results) rather than erroring. The fallback reason names the disabled
+-- flag. The serdes covered here (Concat, SortArray, TruncDate) exercise the shared routing in
+-- QueryPlanSerde across three distinct serdes.
+
+-- Config: spark.comet.exec.scalaUDF.codegen.enabled=false
+
+statement
+CREATE TABLE test_dispatch_disabled_array(c1 array<int>, c2 array<int>) USING parquet
+
+statement
+INSERT INTO test_dispatch_disabled_array VALUES (array(0, 1), array(2, 3)), (array(1, 2), array(3, 4))
+
+-- Concat over array (non-string) children
+query expect_fallback(spark.comet.exec.scalaUDF.codegen.enabled=false)
+SELECT concat(c1, c2) FROM test_dispatch_disabled_array
+
+statement
+CREATE TABLE test_dispatch_disabled_binary(c1 binary, c2 binary) USING parquet
+
+statement
+INSERT INTO test_dispatch_disabled_binary VALUES (unhex('0A0B'), unhex('0C0D')), (unhex('01'), unhex('02'))
+
+-- Concat over binary (non-string) children
+query expect_fallback(spark.comet.exec.scalaUDF.codegen.enabled=false)
+SELECT concat(c1, c2) FROM test_dispatch_disabled_binary
+
+statement
+CREATE TABLE test_dispatch_disabled_nested(arr array<array<struct<a:int>>>) USING parquet
+
+statement
+INSERT INTO test_dispatch_disabled_nested VALUES
+ (array(array(named_struct('a', 2)), array(named_struct('a', 1)))),
+ (NULL)
+
+-- SortArray over nested arrays with Struct children
+query expect_fallback(spark.comet.exec.scalaUDF.codegen.enabled=false)
+SELECT sort_array(arr) FROM test_dispatch_disabled_nested
+
+statement
+CREATE TABLE test_dispatch_disabled_date(d date) USING parquet
+
+statement
+INSERT INTO test_dispatch_disabled_date VALUES (date('2024-06-15')), (NULL)
+
+-- TruncDate with a format outside the native set
+query expect_fallback(spark.comet.exec.scalaUDF.codegen.enabled=false)
+SELECT trunc(d, 'day') FROM test_dispatch_disabled_date
diff --git a/spark/src/test/resources/sql-tests/expressions/string/concat.sql b/spark/src/test/resources/sql-tests/expressions/string/concat.sql
index 3fa2bb65d0..0f299f66a8 100644
--- a/spark/src/test/resources/sql-tests/expressions/string/concat.sql
+++ b/spark/src/test/resources/sql-tests/expressions/string/concat.sql
@@ -57,17 +57,20 @@ SELECT concat('hello', ' ', 'world'), concat('', '', ''), concat(NULL, 'b', 'c')
statement
CREATE TABLE test_concat_binary USING parquet AS SELECT cast(uuid() as binary) c1, cast(uuid() as binary) c2, cast(uuid() as binary) c3, cast(uuid() as binary) c4, cast(null as binary) c5 FROM range(10)
-query expect_fallback(CONCAT supports only string input parameters)
+-- Concat mixes in CodegenDispatchFallback, so non-string (binary) children have no native path
+-- but route through the JVM codegen dispatcher (Spark's own Concat.doGenCode inside the Comet
+-- pipeline) and stay native while matching Spark exactly.
+query
SELECT concat(c1, c2) AS x FROM test_concat_binary
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(c1, c1) AS x FROM test_concat_binary
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(c1, c2, c3) AS x FROM test_concat_binary
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(c1, c2, c3, c5) AS x FROM test_concat_binary
-query expect_fallback(CONCAT supports only string input parameters)
+query
SELECT concat(concat(c1, c2, c3), concat(c1, c3)) AS x FROM test_concat_binary
diff --git a/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala
index 35ceb5c47f..90d6992047 100644
--- a/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometTemporalExpressionSuite.scala
@@ -56,10 +56,11 @@ class CometTemporalExpressionSuite extends CometTestBase with AdaptiveSparkPlanH
checkSparkAnswerAndOperator(s"SELECT c0, trunc(c0, '$format') from tbl order by c0, c1")
}
for (format <- unsupportedFormats) {
- // Comet should fall back to Spark for unsupported or invalid formats
- checkSparkAnswerAndFallbackReason(
- s"SELECT c0, trunc(c0, '$format') from tbl order by c0, c1",
- s"Format $format is not supported")
+ // Formats outside the native set have no native path, but TruncDate mixes in
+ // CodegenDispatchFallback, so they route through the JVM codegen dispatcher (Spark's own
+ // TruncDate.doGenCode inside the Comet pipeline) and stay native while matching Spark
+ // exactly (an invalid format yields null in both engines).
+ checkSparkAnswerAndOperator(s"SELECT c0, trunc(c0, '$format') from tbl order by c0, c1")
}
// Non-literal format strings are Incompatible on the native path, so Comet routes them
@@ -80,10 +81,11 @@ class CometTemporalExpressionSuite extends CometTestBase with AdaptiveSparkPlanH
checkSparkAnswerAndOperator(s"SELECT c0, date_trunc('$format', c0) from tbl order by c0")
}
for (format <- unsupportedFormats) {
- // Comet should fall back to Spark for unsupported or invalid formats
- checkSparkAnswerAndFallbackReason(
- s"SELECT c0, date_trunc('$format', c0) from tbl order by c0",
- s"Format $format is not supported")
+ // Formats outside the native set have no native path, but TruncTimestamp mixes in
+ // CodegenDispatchFallback, so they route through the JVM codegen dispatcher (Spark's own
+ // TruncTimestamp.doGenCode inside the Comet pipeline) and stay native while matching
+ // Spark exactly (an invalid format yields null in both engines).
+ checkSparkAnswerAndOperator(s"SELECT c0, date_trunc('$format', c0) from tbl order by c0")
}
// Non-literal format strings are Incompatible on the native path, so Comet routes them
// through the codegen dispatcher and still executes natively.
@@ -107,10 +109,12 @@ class CometTemporalExpressionSuite extends CometTestBase with AdaptiveSparkPlanH
s"SELECT c0, date_trunc('$format', c0) from tbl order by c0")
}
for (format <- unsupportedFormats) {
- // Comet should fall back to Spark for unsupported or invalid formats
- checkSparkAnswerAndFallbackReason(
- s"SELECT c0, date_trunc('$format', c0) from tbl order by c0",
- s"Format $format is not supported")
+ // Formats outside the native set have no native path, but TruncTimestamp mixes in
+ // CodegenDispatchFallback, so they route through the JVM codegen dispatcher (Spark's
+ // own TruncTimestamp.doGenCode inside the Comet pipeline) and stay native while
+ // matching Spark exactly (an invalid format yields null in both engines).
+ checkSparkAnswerAndOperator(
+ s"SELECT c0, date_trunc('$format', c0) from tbl order by c0")
}
// Non-literal format strings are Incompatible on the native path, so Comet routes them
// through the codegen dispatcher and still executes natively.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org