You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/08 16:10:58 UTC
(datafusion-comet) 01/01: test: add optimizer controls and config isolation to SQL fixtures (#6731)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6731-7a1c042f37c0a4e0cad97ba16aaf558ad30dcb69
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 46d6d7c403c4f5339d1034e1981414964784d603
Author: KUAN-HAO HUANG <10...@users.noreply.github.com>
AuthorDate: Thu Oct 8 16:10:37 2026 +0000
test: add optimizer controls and config isolation to SQL fixtures (#6731)
---
docs/source/contributor-guide/sql-file-tests.md | 76 +++++-
.../array/folded_array_map_literals.sql | 33 +++
.../expressions/math/abs_null_literal.sql | 29 +++
.../apache/comet/CometArrayExpressionSuite.scala | 21 +-
.../org/apache/comet/CometSqlFileTestSuite.scala | 278 ++++++++++++++++++---
.../scala/org/apache/comet/SqlFileTestParser.scala | 52 +++-
.../org/apache/comet/SqlFileTestParserSuite.scala | 79 ++++++
7 files changed, 508 insertions(+), 60 deletions(-)
diff --git a/docs/source/contributor-guide/sql-file-tests.md b/docs/source/contributor-guide/sql-file-tests.md
index 57cc717547..62bff931bb 100644
--- a/docs/source/contributor-guide/sql-file-tests.md
+++ b/docs/source/contributor-guide/sql-file-tests.md
@@ -98,6 +98,9 @@ Sets a Spark SQL config for all queries in the file.
-- Config: spark.sql.ansi.enabled=true
```
+The first `=` separates the key from the value. Values containing `=`, such as base64
+encryption keys, are preserved. An empty value is allowed; the key must be non-empty.
+
#### `ConfigMatrix`
Runs the entire file once per combination of values. Multiple `ConfigMatrix` lines produce a
@@ -117,6 +120,47 @@ sql-file: expressions/conditional/in_set.sql [spark.sql.optimizer.inSetConversio
Only add a `ConfigMatrix` directive when there is a real reason to run the test under
multiple configurations. Do not add `ConfigMatrix` directives speculatively.
+Values are comma-separated and must be non-empty. As with `Config`, any `=` within a
+value is preserved. A matrix value overrides a `Config` value for the same key.
+
+#### `ExcludeRules`
+
+Adds fully qualified optimizer rule names to the exclusion list. Constant folding remains
+disabled by default.
+
+```sql
+-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.NullPropagation
+-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation
+```
+
+Repeated lines are combined, and duplicate rules are removed. Rules declared through
+`Config: spark.sql.optimizer.excludedRules=...` are also retained; a matrix selects the
+value of that config before `ExcludeRules` and the default exclusion are added. Unknown
+rules and rules Spark does not permit excluding fail the fixture rather than silently
+leaving the optimizer unchanged.
+
+Exclude `NullPropagation` when the test needs a NULL argument to reach an expression.
+Disabling ConstantFolding alone does not prevent Spark from replacing `abs(NULL)` with a
+NULL literal.
+
+#### `ConstantFolding`
+
+Lets Spark fold constants for a fixture that tests folded complex literals.
+
+```sql
+-- ConstantFolding: enabled
+```
+
+`disabled` is the default. With `enabled`, an all-literal `array(...)`, `map(...)` or
+`named_struct(...)` can become a complex `Literal` before Comet sees it. This exercises
+Comet's literal handling, rather than its expression constructors. Ordinary all-literal
+function queries may instead fold away entirely; use column inputs when those functions
+must execute in Comet. Combining `enabled` with an explicit ConstantFolding exclusion
+is rejected as contradictory. Repeated `ConstantFolding` directives must agree.
+
+Malformed configuration directives fail that fixture with its file path and source line
+instead of being treated as ordinary comments. Other fixtures still run.
+
#### `MinSparkVersion`
Skips the file when running on a Spark version older than the specified version.
@@ -148,6 +192,34 @@ statement
INSERT INTO my_table VALUES (1, 2.0), (3, 4.0), (NULL, NULL)
```
+Use `SET` statements to change SQL configs between queries:
+
+```sql
+-- Starts with the fixture default, ANSI disabled.
+query
+SELECT CAST(v AS INT) FROM my_strings
+
+statement
+SET spark.sql.ansi.enabled=true
+
+query expect_error(CAST_INVALID_INPUT)
+SELECT CAST(v AS INT) FROM my_strings
+```
+
+All SQLConf changes are restored when the fixture finishes, including on failure and
+between matrix combinations. Keys changed by `SET` need not also appear in `Config`.
+`RESET` is restored too, but it removes the selected setting for the remaining queries;
+`RESET` without a key also resets the harness's ANSI and optimizer settings. A mid-file
+`SET spark.sql.optimizer.excludedRules=...` replaces the whole exclusion list, including
+the default ConstantFolding exclusion.
+
+Enabling the codegen dispatcher through `SET` still requires a non-error sentinel when
+the file contains `expect_error`, just as enabling it through `Config` does. Run that
+sentinel under the same settings as the error query.
+
+This isolation covers SQL configs, not `USE`, temporary views or functions, or SQL
+session variables. Manage those separately.
+
### Queries
A `query` block executes a SELECT and compares results between Spark and Comet. The query
@@ -345,7 +417,9 @@ planning, so Comet would never see them. However, `CometSqlFileTestSuite` automa
disables constant folding (by excluding `ConstantFolding` from the optimizer rules), so
all-literal queries are evaluated by Comet's native engine. This means you can use the
default `query` mode for all-literal cases and they will be tested natively just like
-column-based queries.
+column-based queries, subject to other optimizer rewrites such as `NullPropagation` and
+Comet's own plan-time literal handling. `ConstantFolding: enabled` opts out of the default
+exclusion for fixtures that deliberately test folded literals.
#### Cover edge cases
diff --git a/spark/src/test/resources/sql-tests/expressions/array/folded_array_map_literals.sql b/spark/src/test/resources/sql-tests/expressions/array/folded_array_map_literals.sql
new file mode 100644
index 0000000000..f63cee958d
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/array/folded_array_map_literals.sql
@@ -0,0 +1,33 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ConstantFolding: enabled
+
+statement
+CREATE TABLE folded_array_map_literals(id INT, value BIGINT) USING parquet
+
+statement
+INSERT INTO folded_array_map_literals SELECT CAST(id AS INT), id FROM range(0, 3, 1, 1)
+
+-- The whole array folds to one ArrayType(MapType(...), containsNull=true) literal.
+-- The NULL element and the rebuilt populated map must retain identical map types.
+-- Three rows in one file exercise broadcasting the folded literal across a batch.
+query
+SELECT array(map(1, 2), NULL) AS arr FROM folded_array_map_literals
+
+query
+SELECT array(NULL, map(1, 2)) AS arr FROM folded_array_map_literals
diff --git a/spark/src/test/resources/sql-tests/expressions/math/abs_null_literal.sql b/spark/src/test/resources/sql-tests/expressions/math/abs_null_literal.sql
new file mode 100644
index 0000000000..1f8a03b092
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/math/abs_null_literal.sql
@@ -0,0 +1,29 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements. See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership. The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied. See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.NullPropagation
+
+statement
+CREATE TABLE abs_null_literal(v INT) USING parquet
+
+statement
+INSERT INTO abs_null_literal VALUES (1), (-2), (NULL)
+
+-- Excluding ConstantFolding alone is insufficient: NullPropagation would replace
+-- abs(NULL) with a NULL literal. Pin abs itself so that such a rewrite fails this test.
+query expect_native(abs)
+SELECT abs(CAST(NULL AS INT)) FROM abs_null_literal
diff --git a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala
index 5cbaf53831..c464746c8c 100644
--- a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala
@@ -1379,12 +1379,10 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp
}
}
- // The tests below deliberately live in this suite, not a `sql-tests` fixture: constant folding is
- // enabled by default here, so each `map(...)` collapses to a MapType Literal and the outer
- // `array(...)` reaches `CometCreateArray` with folded-Literal children, which `CometLiteral`
- // rebuilds as an equivalent `CreateMap` of primitive literals -- the folded-literal expansion path
- // under review. `CometSqlFileTestSuite` force-disables `ConstantFolding`, so an equivalent SQL
- // fixture would only exercise the constructor path (which `create_array.sql` already covers).
+ // These tests exercise folded complex literals, including CometLiteral's expansion to native
+ // constructors. SQL fixtures need `-- ConstantFolding: enabled` to reach that same path.
+ // folded_array_map_literals.sql covers the mixed map/NULL array; the remaining conversions are
+ // tracked in #6628.
test("array of folded map literals with array values (multirow)") {
withParquetTable((0 until 3).map(i => (i, i.toLong)), "tbl") {
checkSparkAnswerAndOperator(
@@ -1432,17 +1430,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp
}
}
- // A NULL element sits next to a populated one inside a single folded
- // `ArrayType(MapType(IntegerType, IntegerType, false), containsNull = true)` literal. The null
- // slot serializes as a typed null literal carrying the declared map type, so the rebuilt sibling
- // has to report that same type for `make_array` to accept the pair.
- test("folded array mixing a map literal and a NULL element (multirow)") {
- withParquetTable((0 until 3).map(i => (i, i.toLong)), "tbl") {
- checkSparkAnswerAndOperator("SELECT array(map(1, 2), NULL) AS arr FROM tbl")
- checkSparkAnswerAndOperator("SELECT array(NULL, map(1, 2)) AS arr FROM tbl")
- }
- }
-
// A map value that is itself a map or an array of maps is built by Spark's own generated code
// inside `CometCreateMap`, so it keeps its declared nullability all the way to the consumer.
test("folded map literals with nested map values (multirow)") {
diff --git a/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala b/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala
index b1770b72bf..72ffb23eba 100644
--- a/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala
@@ -21,9 +21,13 @@ package org.apache.comet
import java.io.File
+import scala.util.{Failure, Success, Try}
+
import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.catalyst.expressions.{Abs, CreateArray, Literal}
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.ArrayType
class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
@@ -87,15 +91,56 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
// Disable constant folding so that literal expressions are evaluated by Comet's
// native engine rather than being folded away by Spark's optimizer at plan time.
- private val constantFoldingExcluded = Seq(
- "spark.sql.optimizer.excludedRules" ->
- "org.apache.spark.sql.catalyst.optimizer.ConstantFolding")
+ private val constantFoldingRule = "org.apache.spark.sql.catalyst.optimizer.ConstantFolding"
// Most SQL fixtures here predate Spark 4 ANSI default and expect non-ANSI semantics
// (silent overflow/null on bad input). Individual files can opt in via their own
// --CONFIG line, which appears later in the pair list and wins.
private val ansiDisabled = Seq(SQLConf.ANSI_ENABLED.key -> "false")
+ /** Apply file settings and restore every SQLConf change, including mid-file SET and RESET. */
+ private def withFixtureConfigs(file: SqlTestFile)(body: => Unit): Unit = {
+ val conf = spark.sessionState.conf
+ val original = conf.getAllConfs
+ val optimizerKey = SQLConf.OPTIMIZER_EXCLUDED_RULES.key
+ // Matrix settings are appended to configs, so the last value for a key wins as before.
+ val configuredRules = file.configs.reverse
+ .collectFirst { case (`optimizerKey`, value) =>
+ value
+ }
+ .toSeq
+ .flatMap(_.split(","))
+ val rules = (configuredRules ++ file.excludedRules ++
+ (if (file.constantFoldingEnabled) Seq.empty else Seq(constantFoldingRule)))
+ .map(_.trim)
+ .filter(_.nonEmpty)
+ .distinct
+ require(
+ !file.constantFoldingEnabled || !rules.contains(constantFoldingRule),
+ "ConstantFolding: enabled conflicts with an explicit ConstantFolding exclusion")
+ // Spark silently ignores unknown and non-excludable rules. A fixture must fail instead of
+ // claiming to test a path that its optimizer settings never reached.
+ val optimizer = spark.sessionState.optimizer
+ val knownRules = optimizer.defaultBatches.flatMap(_.rules).map(_.ruleName).toSet
+ rules.foreach { rule =>
+ require(knownRules.contains(rule), s"Unknown optimizer rule: $rule")
+ require(
+ !optimizer.nonExcludableRules.contains(rule),
+ s"Cannot exclude optimizer rule: $rule")
+ }
+ val configs = ansiDisabled ++ file.configs.filterNot(_._1 == optimizerKey) :+
+ (optimizerKey -> rules.mkString(","))
+ try {
+ withSQLConf(configs: _*)(body)
+ } finally {
+ val current = conf.getAllConfs
+ (current.keySet -- original.keySet).foreach(conf.unsetConf)
+ original.foreach { case (key, value) =>
+ if (current.get(key) != Some(value)) conf.setConfString(key, value)
+ }
+ }
+ }
+
/**
* Pin the sentinel-query convention for fixtures that route an `expect_error` through the
* codegen dispatcher. See [[ExpectError]] for the failure mode this guards against.
@@ -103,9 +148,9 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
private def requireSentinelForCodegenExpectError(
relativePath: String,
file: SqlTestFile): Unit = {
- val codegenFlagOn = file.configs.exists { case (k, v) =>
- k == CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key && v.equalsIgnoreCase("true")
- }
+ val codegenFlagOn = spark.sessionState.conf.getAllConfs
+ .get(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key)
+ .exists(_.equalsIgnoreCase("true"))
if (!codegenFlagOn) return
val hasExpectError = file.records.exists {
case SqlQuery(_, _: ExpectError, _) => true
@@ -132,9 +177,7 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
private def runTestFile(relativePath: String, file: SqlTestFile): Unit = {
- requireSentinelForCodegenExpectError(relativePath, file)
- val allConfigs = ansiDisabled ++ file.configs ++ constantFoldingExcluded
- withSQLConf(allConfigs: _*) {
+ withFixtureConfigs(file) {
withTable(file.tables: _*) {
file.records.foreach {
case SqlStatement(sql, line) =>
@@ -167,6 +210,8 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
case Ignore(reason) =>
logInfo(s"IGNORED query ($reason): $sql")
case ExpectError(pattern) =>
+ // Check the effective setting after any mid-file SET or RESET.
+ requireSentinelForCodegenExpectError(relativePath, file)
val (sparkError, cometError) = checkSparkAnswerMaybeThrows(spark.sql(sql))
assert(
sparkError.isDefined,
@@ -192,39 +237,198 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}
}
+ test("SQL fixture optimizer directives control the optimized expression") {
+ val default = SqlFileTestParser.parse(Seq.empty)
+ withFixtureConfigs(default) {
+ val plan = spark.sql("SELECT array(1, 2)").queryExecution.optimizedPlan
+ assert(plan.expressions.exists(_.exists(_.isInstanceOf[CreateArray])))
+ val nullPlan = spark.sql("SELECT abs(CAST(NULL AS INT))").queryExecution.optimizedPlan
+ assert(!nullPlan.expressions.exists(_.exists(_.isInstanceOf[Abs])))
+ }
+ val folded = SqlFileTestParser.parse(Seq("-- ConstantFolding: enabled"))
+ withFixtureConfigs(folded) {
+ val plan = spark.sql("SELECT array(1, 2)").queryExecution.optimizedPlan
+ assert(plan.expressions.exists(_.exists {
+ case Literal(_, _: ArrayType) => true
+ case _ => false
+ }))
+ assert(!plan.expressions.exists(_.exists(_.isInstanceOf[CreateArray])))
+ }
+ val excluded = SqlFileTestParser.parse(
+ Seq("-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.NullPropagation"))
+ withFixtureConfigs(excluded) {
+ val plan = spark.sql("SELECT abs(CAST(NULL AS INT))").queryExecution.optimizedPlan
+ assert(plan.expressions.exists(_.exists(_.isInstanceOf[Abs])))
+ assert(spark.conf.get(SQLConf.OPTIMIZER_EXCLUDED_RULES.key).contains(constantFoldingRule))
+ }
+ }
+
+ test("SQL fixture optimizer Config and matrix settings retain precedence") {
+ val optimizerKey = SQLConf.OPTIMIZER_EXCLUDED_RULES.key
+ val nullPropagation = "org.apache.spark.sql.catalyst.optimizer.NullPropagation"
+ val eliminateSorts = "org.apache.spark.sql.catalyst.optimizer.EliminateSorts"
+ val file = SqlFileTestParser.parse(
+ Seq(
+ s"-- Config: $optimizerKey=$nullPropagation",
+ s"-- ConfigMatrix: $optimizerKey=$eliminateSorts,$nullPropagation",
+ s"-- ExcludeRules: $nullPropagation, $nullPropagation"))
+ val original = spark.sessionState.conf.getAllConfs
+ configMatrix(file.configMatrix).foreach { matrix =>
+ withFixtureConfigs(file.copy(configs = file.configs ++ matrix)) {
+ val selected = matrix.head._2
+ assert(
+ spark.conf.get(optimizerKey).split(",").toSeq ==
+ Seq(selected, nullPropagation, constantFoldingRule).distinct)
+ }
+ assert(spark.sessionState.conf.getAllConfs == original)
+ }
+ }
+
+ test("SQL fixture optimizer directives reject ineffective exclusions") {
+ Seq(
+ "org.apache.spark.sql.catalyst.optimizer.NoSuchRule" -> "Unknown optimizer rule",
+ spark.sessionState.optimizer.nonExcludableRules.head -> "Cannot exclude optimizer rule")
+ .foreach { case (rule, message) =>
+ val file = SqlFileTestParser.parse(Seq(s"-- ExcludeRules: $rule"))
+ assert(
+ intercept[IllegalArgumentException](withFixtureConfigs(file) {}).getMessage
+ .contains(message))
+ }
+ val conflict = SqlFileTestParser.parse(
+ Seq("-- ConstantFolding: enabled", s"-- ExcludeRules: $constantFoldingRule"))
+ assert(
+ intercept[IllegalArgumentException](withFixtureConfigs(conflict) {}).getMessage
+ .contains("conflicts"))
+ }
+
+ test("SQL fixture SET and RESET restore existing and previously unset keys") {
+ val key = "spark.comet.sqlFixture.isolation"
+ val original = spark.sessionState.conf.getAllConfs
+ assert(!original.contains(key))
+ val file = SqlFileTestParser.parse(Seq("statement", s"SET $key=first"))
+ runTestFile("set-isolation", file)
+ assert(spark.sessionState.conf.getAllConfs == original)
+ withSQLConf(key -> "original") {
+ val before = spark.sessionState.conf.getAllConfs
+ runTestFile("set-existing", file)
+ assert(spark.sessionState.conf.getAllConfs == before)
+ runTestFile("reset-existing", SqlFileTestParser.parse(Seq("statement", s"RESET $key")))
+ assert(spark.sessionState.conf.getAllConfs == before)
+ runTestFile("reset-all", SqlFileTestParser.parse(Seq("statement", "RESET")))
+ assert(spark.sessionState.conf.getAllConfs == before)
+ }
+ }
+
+ test("SQL fixture settings are restored after a failing statement or query") {
+ val key = "spark.comet.sqlFixture.isolation"
+ Seq("statement", "query").foreach { directive =>
+ val original = spark.sessionState.conf.getAllConfs
+ val file = SqlFileTestParser.parse(
+ Seq(
+ "statement",
+ s"SET $key=changed",
+ "",
+ directive,
+ "SELECT * FROM missing_sql_fixture_isolation_table"))
+ intercept[RuntimeException](runTestFile("failure-isolation", file))
+ assert(spark.sessionState.conf.getAllConfs == original)
+ }
+ }
+
+ test("SQL fixture SET changes take effect between queries") {
+ val file = SqlFileTestParser.parse(
+ Seq(
+ "statement",
+ "CREATE TABLE sql_fixture_set(v STRING) USING parquet",
+ "",
+ "statement",
+ "INSERT INTO sql_fixture_set VALUES ('invalid'), ('1')",
+ "",
+ "query",
+ "SELECT CAST(v AS INT) FROM sql_fixture_set",
+ "",
+ "statement",
+ "SET spark.sql.ansi.enabled=true",
+ "",
+ "query expect_error(CAST_INVALID_INPUT)",
+ "SELECT CAST(v AS INT) FROM sql_fixture_set"))
+ val original = spark.sessionState.conf.getAllConfs
+ runTestFile("set-between-queries", file)
+ assert(spark.sessionState.conf.getAllConfs == original)
+ }
+
+ test("SQL fixture SET cannot bypass the dispatcher error sentinel") {
+ val file = SqlFileTestParser.parse(
+ Seq(
+ "-- Config: spark.sql.ansi.enabled=true",
+ s"-- Config: ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=false",
+ "statement",
+ s"SET ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=true",
+ "",
+ "query expect_error(CAST_INVALID_INPUT)",
+ "SELECT CAST('invalid' AS INT)"))
+ val original = spark.sessionState.conf.getAllConfs
+ val error = intercept[Exception](runTestFile("set-dispatcher-sentinel", file))
+ assert(error.getMessage.contains("missing a non-error"))
+ assert(spark.sessionState.conf.getAllConfs == original)
+ }
+
+ test("SQL fixture dispatcher sentinel follows SET before an error query") {
+ val file = SqlFileTestParser.parse(
+ Seq(
+ "-- Config: spark.sql.ansi.enabled=true",
+ s"-- Config: ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=true",
+ "statement",
+ s"SET ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=false",
+ "",
+ "query expect_error(CAST_INVALID_INPUT)",
+ "SELECT CAST('invalid' AS INT)"))
+ val original = spark.sessionState.conf.getAllConfs
+ runTestFile("set-disable-dispatcher", file)
+ assert(spark.sessionState.conf.getAllConfs == original)
+ }
+
// Discover and register all .sql test files
discoverTestFiles(testResourceDir).foreach { file =>
val relativePath = testResourceDir.toURI.relativize(file.toURI).getPath
- val parsed = SqlFileTestParser.parse(file)
- val combinations = configMatrix(parsed.configMatrix)
-
- // Skip tests that fall outside the file's declared Spark version range.
- val skip = skipReason(parsed)
-
- if (combinations.size <= 1) {
- // No matrix or single combination
- test(s"sql-file: $relativePath") {
- skip match {
- case Some(reason) =>
- logInfo(s"SKIPPED ($reason): $relativePath")
- case None =>
- val effectiveConfigs = parsed.configs ++ combinations.headOption.getOrElse(Seq.empty)
- runTestFile(relativePath, parsed.copy(configs = effectiveConfigs))
- }
- }
- } else {
- // Multiple combinations: generate one test per combination
- combinations.foreach { matrixConfigs =>
- val label = matrixConfigs.map { case (k, v) => s"$k=$v" }.mkString(", ")
- test(s"sql-file: $relativePath [$label]") {
- skip match {
- case Some(reason) =>
- logInfo(s"SKIPPED ($reason): $relativePath")
- case None =>
- runTestFile(relativePath, parsed.copy(configs = parsed.configs ++ matrixConfigs))
+ Try(SqlFileTestParser.parse(file)) match {
+ case Failure(error) =>
+ // A malformed file must not prevent every other fixture from being registered.
+ test(s"sql-file: $relativePath") { throw error }
+ case Success(parsed) =>
+ val combinations = configMatrix(parsed.configMatrix)
+
+ // Skip tests that fall outside the file's declared Spark version range.
+ val skip = skipReason(parsed)
+
+ if (combinations.size <= 1) {
+ // No matrix or single combination
+ test(s"sql-file: $relativePath") {
+ skip match {
+ case Some(reason) =>
+ logInfo(s"SKIPPED ($reason): $relativePath")
+ case None =>
+ val effectiveConfigs =
+ parsed.configs ++ combinations.headOption.getOrElse(Seq.empty)
+ runTestFile(relativePath, parsed.copy(configs = effectiveConfigs))
+ }
+ }
+ } else {
+ // Multiple combinations: generate one test per combination
+ combinations.foreach { matrixConfigs =>
+ val label = matrixConfigs.map { case (k, v) => s"$k=$v" }.mkString(", ")
+ test(s"sql-file: $relativePath [$label]") {
+ skip match {
+ case Some(reason) =>
+ logInfo(s"SKIPPED ($reason): $relativePath")
+ case None =>
+ runTestFile(
+ relativePath,
+ parsed.copy(configs = parsed.configs ++ matrixConfigs))
+ }
+ }
}
}
- }
}
}
}
diff --git a/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala b/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala
index 2b7c573bb1..f7e653c2e1 100644
--- a/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala
+++ b/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala
@@ -104,6 +104,10 @@ case class ExpectError(pattern: String) extends QueryAssertionMode
* Optional maximum Spark version this test applies to (e.g. "3.4"). The test is skipped on
* newer versions. Useful for paired fixtures where each version range has its own expected
* error class or output format.
+ * @param excludedRules
+ * Additional optimizer rule names to exclude, alongside the default ConstantFolding exclusion.
+ * @param constantFoldingEnabled
+ * Whether to let Spark fold constants before Comet sees the query.
*/
case class SqlTestFile(
configs: Seq[(String, String)],
@@ -111,12 +115,18 @@ case class SqlTestFile(
records: Seq[SqlTestRecord],
tables: Seq[String],
minSparkVersion: Option[String] = None,
- maxSparkVersion: Option[String] = None)
+ maxSparkVersion: Option[String] = None,
+ excludedRules: Seq[String] = Seq.empty,
+ constantFoldingEnabled: Boolean = false)
object SqlFileTestParser {
- private val ConfigPattern = """--\s*Config:\s*(.+)=(.+)""".r
- private val ConfigMatrixPattern = """--\s*ConfigMatrix:\s*(.+)=(.+)""".r
+ private val ConfigPattern = """--\s*Config:\s*([^=]+)=(.*)""".r
+ private val ConfigMatrixPattern = """--\s*ConfigMatrix:\s*([^=]+)=(.*)""".r
+ private val ExcludeRulesPattern = """--\s*ExcludeRules:\s*(.*)""".r
+ private val ConstantFoldingPattern = """--\s*ConstantFolding:\s*(.*)""".r
+ private val ConfigDirectivePattern =
+ """--\s*(Config|ConfigMatrix|ExcludeRules|ConstantFolding):.*""".r
private val MinSparkVersionPattern = """--\s*MinSparkVersion:\s*(.+)""".r
private val MaxSparkVersionPattern = """--\s*MaxSparkVersion:\s*(.+)""".r
private val CreateTablePattern = """(?i)CREATE\s+TABLE\s+(\w+)""".r.unanchored
@@ -125,6 +135,9 @@ object SqlFileTestParser {
val source = Source.fromFile(file, "UTF-8")
try {
parse(source.getLines().toSeq)
+ } catch {
+ case e: IllegalArgumentException =>
+ throw new IllegalArgumentException(s"${file.getPath}: ${e.getMessage}", e)
} finally {
source.close()
}
@@ -135,6 +148,8 @@ object SqlFileTestParser {
var configMatrix = Seq.empty[(String, Seq[String])]
var minSparkVersion: Option[String] = None
var maxSparkVersion: Option[String] = None
+ var excludedRules = Seq.empty[String]
+ var constantFoldingEnabled: Option[Boolean] = None
val records = Seq.newBuilder[SqlTestRecord]
val tables = Seq.newBuilder[String]
@@ -144,13 +159,38 @@ object SqlFileTestParser {
line match {
case ConfigPattern(key, value) =>
+ require(key.trim.nonEmpty, s"Empty Config key at line ${lineIdx + 1}")
configs :+= (key.trim -> value.trim)
lineIdx += 1
case ConfigMatrixPattern(key, values) =>
- configMatrix :+= (key.trim -> values.split(",").map(_.trim).toSeq)
+ require(key.trim.nonEmpty, s"Empty ConfigMatrix key at line ${lineIdx + 1}")
+ val choices = values.split(",", -1).map(_.trim).toSeq
+ require(choices.forall(_.nonEmpty), s"Empty ConfigMatrix value at line ${lineIdx + 1}")
+ configMatrix :+= (key.trim -> choices)
lineIdx += 1
+ case ExcludeRulesPattern(rules) =>
+ val names = splitNames(rules)
+ require(names.nonEmpty, s"Empty ExcludeRules at line ${lineIdx + 1}")
+ excludedRules ++= names
+ lineIdx += 1
+
+ case ConstantFoldingPattern(mode) =>
+ require(
+ mode == "enabled" || mode == "disabled",
+ s"Invalid ConstantFolding mode '$mode' at line ${lineIdx + 1}: " +
+ "expected enabled or disabled")
+ val enabled = mode == "enabled"
+ require(
+ constantFoldingEnabled.forall(_ == enabled),
+ s"Conflicting ConstantFolding directives at line ${lineIdx + 1}")
+ constantFoldingEnabled = Some(enabled)
+ lineIdx += 1
+
+ case ConfigDirectivePattern(name) =>
+ throw new IllegalArgumentException(s"Malformed $name directive at line ${lineIdx + 1}")
+
case MinSparkVersionPattern(version) =>
minSparkVersion = Some(version.trim)
lineIdx += 1
@@ -188,7 +228,9 @@ object SqlFileTestParser {
records.result(),
tables.result(),
minSparkVersion,
- maxSparkVersion)
+ maxSparkVersion,
+ excludedRules.distinct,
+ constantFoldingEnabled.getOrElse(false))
}
private val FallbackPattern = """query\s+expect_fallback\((.+)\)""".r
diff --git a/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala b/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala
index c356c3c37a..0d107395ba 100644
--- a/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala
@@ -20,6 +20,8 @@
package org.apache.comet
import java.io.File
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
import org.scalatest.funsuite.AnyFunSuite
@@ -78,6 +80,83 @@ class SqlFileTestParserSuite extends AnyFunSuite {
}
}
+ test("configuration values preserve equals signs and empty Config values") {
+ val parsed = SqlFileTestParser.parse(
+ Seq(
+ "-- Config: fixture.key = MDEyMzQ1Njc4OTAxMjM0NQ==",
+ "-- Config: fixture.other = a=b=c",
+ "-- Config: fixture.empty=",
+ "-- ConfigMatrix: fixture.matrix = first==, second=part="))
+ assert(
+ parsed.configs == Seq(
+ "fixture.key" -> "MDEyMzQ1Njc4OTAxMjM0NQ==",
+ "fixture.other" -> "a=b=c",
+ "fixture.empty" -> ""))
+ assert(parsed.configMatrix == Seq("fixture.matrix" -> Seq("first==", "second=part=")))
+ }
+
+ test("optimizer directives preserve defaults and combine repeated exclusions") {
+ val defaults = SqlFileTestParser.parse(Seq.empty)
+ assert(!defaults.constantFoldingEnabled)
+ assert(defaults.excludedRules.isEmpty)
+ val parsed = SqlFileTestParser.parse(
+ Seq(
+ "-- ConstantFolding: enabled",
+ "-- ExcludeRules: first.Rule, second.Rule,",
+ "-- ExcludeRules: second.Rule, third.Rule"))
+ assert(parsed.constantFoldingEnabled)
+ assert(parsed.excludedRules == Seq("first.Rule", "second.Rule", "third.Rule"))
+ assert(!SqlFileTestParser.parse(Seq("-- ConstantFolding: disabled")).constantFoldingEnabled)
+ }
+
+ test("malformed configuration directives fail with their source line") {
+ Seq(
+ "-- Config: no_assignment",
+ "-- Config: =value",
+ "-- ConfigMatrix: no_assignment",
+ "-- ConfigMatrix: key=",
+ "-- ConfigMatrix: key=first,,second",
+ "-- ConfigMatrix: key=first,",
+ "-- ExcludeRules:",
+ "-- ExcludeRules: , ,",
+ "-- ConstantFolding: true").foreach { directive =>
+ val error = intercept[IllegalArgumentException] {
+ SqlFileTestParser.parse(Seq("-- comment", directive))
+ }
+ assert(error.getMessage.contains("line 2"), directive)
+ }
+ // Prose mentioning a directive is still an ordinary comment unless it includes the colon.
+ assert(
+ SqlFileTestParser.parse(Seq("-- ConstantFolding, so literals remain visible")) ==
+ SqlTestFile(Seq.empty, Seq.empty, Seq.empty, Seq.empty))
+ }
+
+ test("repeated ConstantFolding directives must agree") {
+ Seq("enabled", "disabled").foreach { mode =>
+ val directive = s"-- ConstantFolding: $mode"
+ val parsed = SqlFileTestParser.parse(Seq(directive, directive))
+ assert(parsed.constantFoldingEnabled == (mode == "enabled"))
+ val other = if (mode == "enabled") "disabled" else "enabled"
+ val error = intercept[IllegalArgumentException] {
+ SqlFileTestParser.parse(Seq(directive, s"-- ConstantFolding: $other"))
+ }
+ assert(error.getMessage.contains("Conflicting ConstantFolding directives at line 2"))
+ }
+ }
+
+ test("malformed fixture files report their path and source line") {
+ val tempDir = Files.createDirectories(new File(System.getProperty("java.io.tmpdir")).toPath)
+ val path = Files.createTempFile(tempDir, "comet-malformed-fixture-", ".sql")
+ try {
+ Files.write(path, "-- comment\n-- Config: no_assignment".getBytes(StandardCharsets.UTF_8))
+ val error = intercept[IllegalArgumentException](SqlFileTestParser.parse(path.toFile))
+ assert(error.getMessage.contains(path.toString))
+ assert(error.getMessage.contains("line 2"))
+ } finally {
+ Files.delete(path)
+ }
+ }
+
test("statements and queries preserve SQL, source lines and tables for cleanup") {
val lines = Seq(
"-- fixture comment",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org