You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/08 16:10:58 UTC

(datafusion-comet) 01/01: test: add optimizer controls and config isolation to SQL fixtures (#6731)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6731-7a1c042f37c0a4e0cad97ba16aaf558ad30dcb69
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git

commit 46d6d7c403c4f5339d1034e1981414964784d603
Author: KUAN-HAO HUANG <10...@users.noreply.github.com>
AuthorDate: Thu Oct 8 16:10:37 2026 +0000

    test: add optimizer controls and config isolation to SQL fixtures (#6731)
---
 docs/source/contributor-guide/sql-file-tests.md    |  76 +++++-
 .../array/folded_array_map_literals.sql            |  33 +++
 .../expressions/math/abs_null_literal.sql          |  29 +++
 .../apache/comet/CometArrayExpressionSuite.scala   |  21 +-
 .../org/apache/comet/CometSqlFileTestSuite.scala   | 278 ++++++++++++++++++---
 .../scala/org/apache/comet/SqlFileTestParser.scala |  52 +++-
 .../org/apache/comet/SqlFileTestParserSuite.scala  |  79 ++++++
 7 files changed, 508 insertions(+), 60 deletions(-)

diff --git a/docs/source/contributor-guide/sql-file-tests.md b/docs/source/contributor-guide/sql-file-tests.md
index 57cc717547..62bff931bb 100644
--- a/docs/source/contributor-guide/sql-file-tests.md
+++ b/docs/source/contributor-guide/sql-file-tests.md
@@ -98,6 +98,9 @@ Sets a Spark SQL config for all queries in the file.
 -- Config: spark.sql.ansi.enabled=true
 ```
 
+The first `=` separates the key from the value. Values containing `=`, such as base64
+encryption keys, are preserved. An empty value is allowed; the key must be non-empty.
+
 #### `ConfigMatrix`
 
 Runs the entire file once per combination of values. Multiple `ConfigMatrix` lines produce a
@@ -117,6 +120,47 @@ sql-file: expressions/conditional/in_set.sql [spark.sql.optimizer.inSetConversio
 Only add a `ConfigMatrix` directive when there is a real reason to run the test under
 multiple configurations. Do not add `ConfigMatrix` directives speculatively.
 
+Values are comma-separated and must be non-empty. As with `Config`, any `=` within a
+value is preserved. A matrix value overrides a `Config` value for the same key.
+
+#### `ExcludeRules`
+
+Adds fully qualified optimizer rule names to the exclusion list. Constant folding remains
+disabled by default.
+
+```sql
+-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.NullPropagation
+-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation
+```
+
+Repeated lines are combined, and duplicate rules are removed. Rules declared through
+`Config: spark.sql.optimizer.excludedRules=...` are also retained; a matrix selects the
+value of that config before `ExcludeRules` and the default exclusion are added. Unknown
+rules and rules Spark does not permit excluding fail the fixture rather than silently
+leaving the optimizer unchanged.
+
+Exclude `NullPropagation` when the test needs a NULL argument to reach an expression.
+Disabling ConstantFolding alone does not prevent Spark from replacing `abs(NULL)` with a
+NULL literal.
+
+#### `ConstantFolding`
+
+Lets Spark fold constants for a fixture that tests folded complex literals.
+
+```sql
+-- ConstantFolding: enabled
+```
+
+`disabled` is the default. With `enabled`, an all-literal `array(...)`, `map(...)` or
+`named_struct(...)` can become a complex `Literal` before Comet sees it. This exercises
+Comet's literal handling, rather than its expression constructors. Ordinary all-literal
+function queries may instead fold away entirely; use column inputs when those functions
+must execute in Comet. Combining `enabled` with an explicit ConstantFolding exclusion
+is rejected as contradictory. Repeated `ConstantFolding` directives must agree.
+
+Malformed configuration directives fail that fixture with its file path and source line
+instead of being treated as ordinary comments. Other fixtures still run.
+
 #### `MinSparkVersion`
 
 Skips the file when running on a Spark version older than the specified version.
@@ -148,6 +192,34 @@ statement
 INSERT INTO my_table VALUES (1, 2.0), (3, 4.0), (NULL, NULL)
 ```
 
+Use `SET` statements to change SQL configs between queries:
+
+```sql
+-- Starts with the fixture default, ANSI disabled.
+query
+SELECT CAST(v AS INT) FROM my_strings
+
+statement
+SET spark.sql.ansi.enabled=true
+
+query expect_error(CAST_INVALID_INPUT)
+SELECT CAST(v AS INT) FROM my_strings
+```
+
+All SQLConf changes are restored when the fixture finishes, including on failure and
+between matrix combinations. Keys changed by `SET` need not also appear in `Config`.
+`RESET` is restored too, but it removes the selected setting for the remaining queries;
+`RESET` without a key also resets the harness's ANSI and optimizer settings. A mid-file
+`SET spark.sql.optimizer.excludedRules=...` replaces the whole exclusion list, including
+the default ConstantFolding exclusion.
+
+Enabling the codegen dispatcher through `SET` still requires a non-error sentinel when
+the file contains `expect_error`, just as enabling it through `Config` does. Run that
+sentinel under the same settings as the error query.
+
+This isolation covers SQL configs, not `USE`, temporary views or functions, or SQL
+session variables. Manage those separately.
+
 ### Queries
 
 A `query` block executes a SELECT and compares results between Spark and Comet. The query
@@ -345,7 +417,9 @@ planning, so Comet would never see them. However, `CometSqlFileTestSuite` automa
 disables constant folding (by excluding `ConstantFolding` from the optimizer rules), so
 all-literal queries are evaluated by Comet's native engine. This means you can use the
 default `query` mode for all-literal cases and they will be tested natively just like
-column-based queries.
+column-based queries, subject to other optimizer rewrites such as `NullPropagation` and
+Comet's own plan-time literal handling. `ConstantFolding: enabled` opts out of the default
+exclusion for fixtures that deliberately test folded literals.
 
 #### Cover edge cases
 
diff --git a/spark/src/test/resources/sql-tests/expressions/array/folded_array_map_literals.sql b/spark/src/test/resources/sql-tests/expressions/array/folded_array_map_literals.sql
new file mode 100644
index 0000000000..f63cee958d
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/array/folded_array_map_literals.sql
@@ -0,0 +1,33 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ConstantFolding: enabled
+
+statement
+CREATE TABLE folded_array_map_literals(id INT, value BIGINT) USING parquet
+
+statement
+INSERT INTO folded_array_map_literals SELECT CAST(id AS INT), id FROM range(0, 3, 1, 1)
+
+-- The whole array folds to one ArrayType(MapType(...), containsNull=true) literal.
+-- The NULL element and the rebuilt populated map must retain identical map types.
+-- Three rows in one file exercise broadcasting the folded literal across a batch.
+query
+SELECT array(map(1, 2), NULL) AS arr FROM folded_array_map_literals
+
+query
+SELECT array(NULL, map(1, 2)) AS arr FROM folded_array_map_literals
diff --git a/spark/src/test/resources/sql-tests/expressions/math/abs_null_literal.sql b/spark/src/test/resources/sql-tests/expressions/math/abs_null_literal.sql
new file mode 100644
index 0000000000..1f8a03b092
--- /dev/null
+++ b/spark/src/test/resources/sql-tests/expressions/math/abs_null_literal.sql
@@ -0,0 +1,29 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.NullPropagation
+
+statement
+CREATE TABLE abs_null_literal(v INT) USING parquet
+
+statement
+INSERT INTO abs_null_literal VALUES (1), (-2), (NULL)
+
+-- Excluding ConstantFolding alone is insufficient: NullPropagation would replace
+-- abs(NULL) with a NULL literal. Pin abs itself so that such a rewrite fails this test.
+query expect_native(abs)
+SELECT abs(CAST(NULL AS INT)) FROM abs_null_literal
diff --git a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala
index 5cbaf53831..c464746c8c 100644
--- a/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometArrayExpressionSuite.scala
@@ -1379,12 +1379,10 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp
     }
   }
 
-  // The tests below deliberately live in this suite, not a `sql-tests` fixture: constant folding is
-  // enabled by default here, so each `map(...)` collapses to a MapType Literal and the outer
-  // `array(...)` reaches `CometCreateArray` with folded-Literal children, which `CometLiteral`
-  // rebuilds as an equivalent `CreateMap` of primitive literals -- the folded-literal expansion path
-  // under review. `CometSqlFileTestSuite` force-disables `ConstantFolding`, so an equivalent SQL
-  // fixture would only exercise the constructor path (which `create_array.sql` already covers).
+  // These tests exercise folded complex literals, including CometLiteral's expansion to native
+  // constructors. SQL fixtures need `-- ConstantFolding: enabled` to reach that same path.
+  // folded_array_map_literals.sql covers the mixed map/NULL array; the remaining conversions are
+  // tracked in #6628.
   test("array of folded map literals with array values (multirow)") {
     withParquetTable((0 until 3).map(i => (i, i.toLong)), "tbl") {
       checkSparkAnswerAndOperator(
@@ -1432,17 +1430,6 @@ class CometArrayExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelp
     }
   }
 
-  // A NULL element sits next to a populated one inside a single folded
-  // `ArrayType(MapType(IntegerType, IntegerType, false), containsNull = true)` literal. The null
-  // slot serializes as a typed null literal carrying the declared map type, so the rebuilt sibling
-  // has to report that same type for `make_array` to accept the pair.
-  test("folded array mixing a map literal and a NULL element (multirow)") {
-    withParquetTable((0 until 3).map(i => (i, i.toLong)), "tbl") {
-      checkSparkAnswerAndOperator("SELECT array(map(1, 2), NULL) AS arr FROM tbl")
-      checkSparkAnswerAndOperator("SELECT array(NULL, map(1, 2)) AS arr FROM tbl")
-    }
-  }
-
   // A map value that is itself a map or an array of maps is built by Spark's own generated code
   // inside `CometCreateMap`, so it keeps its declared nullability all the way to the consumer.
   test("folded map literals with nested map values (multirow)") {
diff --git a/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala b/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala
index b1770b72bf..72ffb23eba 100644
--- a/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/CometSqlFileTestSuite.scala
@@ -21,9 +21,13 @@ package org.apache.comet
 
 import java.io.File
 
+import scala.util.{Failure, Success, Try}
+
 import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.catalyst.expressions.{Abs, CreateArray, Literal}
 import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
 import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.ArrayType
 
 class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
@@ -87,15 +91,56 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
 
   // Disable constant folding so that literal expressions are evaluated by Comet's
   // native engine rather than being folded away by Spark's optimizer at plan time.
-  private val constantFoldingExcluded = Seq(
-    "spark.sql.optimizer.excludedRules" ->
-      "org.apache.spark.sql.catalyst.optimizer.ConstantFolding")
+  private val constantFoldingRule = "org.apache.spark.sql.catalyst.optimizer.ConstantFolding"
 
   // Most SQL fixtures here predate Spark 4 ANSI default and expect non-ANSI semantics
   // (silent overflow/null on bad input). Individual files can opt in via their own
   // --CONFIG line, which appears later in the pair list and wins.
   private val ansiDisabled = Seq(SQLConf.ANSI_ENABLED.key -> "false")
 
+  /** Apply file settings and restore every SQLConf change, including mid-file SET and RESET. */
+  private def withFixtureConfigs(file: SqlTestFile)(body: => Unit): Unit = {
+    val conf = spark.sessionState.conf
+    val original = conf.getAllConfs
+    val optimizerKey = SQLConf.OPTIMIZER_EXCLUDED_RULES.key
+    // Matrix settings are appended to configs, so the last value for a key wins as before.
+    val configuredRules = file.configs.reverse
+      .collectFirst { case (`optimizerKey`, value) =>
+        value
+      }
+      .toSeq
+      .flatMap(_.split(","))
+    val rules = (configuredRules ++ file.excludedRules ++
+      (if (file.constantFoldingEnabled) Seq.empty else Seq(constantFoldingRule)))
+      .map(_.trim)
+      .filter(_.nonEmpty)
+      .distinct
+    require(
+      !file.constantFoldingEnabled || !rules.contains(constantFoldingRule),
+      "ConstantFolding: enabled conflicts with an explicit ConstantFolding exclusion")
+    // Spark silently ignores unknown and non-excludable rules. A fixture must fail instead of
+    // claiming to test a path that its optimizer settings never reached.
+    val optimizer = spark.sessionState.optimizer
+    val knownRules = optimizer.defaultBatches.flatMap(_.rules).map(_.ruleName).toSet
+    rules.foreach { rule =>
+      require(knownRules.contains(rule), s"Unknown optimizer rule: $rule")
+      require(
+        !optimizer.nonExcludableRules.contains(rule),
+        s"Cannot exclude optimizer rule: $rule")
+    }
+    val configs = ansiDisabled ++ file.configs.filterNot(_._1 == optimizerKey) :+
+      (optimizerKey -> rules.mkString(","))
+    try {
+      withSQLConf(configs: _*)(body)
+    } finally {
+      val current = conf.getAllConfs
+      (current.keySet -- original.keySet).foreach(conf.unsetConf)
+      original.foreach { case (key, value) =>
+        if (current.get(key) != Some(value)) conf.setConfString(key, value)
+      }
+    }
+  }
+
   /**
    * Pin the sentinel-query convention for fixtures that route an `expect_error` through the
    * codegen dispatcher. See [[ExpectError]] for the failure mode this guards against.
@@ -103,9 +148,9 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
   private def requireSentinelForCodegenExpectError(
       relativePath: String,
       file: SqlTestFile): Unit = {
-    val codegenFlagOn = file.configs.exists { case (k, v) =>
-      k == CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key && v.equalsIgnoreCase("true")
-    }
+    val codegenFlagOn = spark.sessionState.conf.getAllConfs
+      .get(CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key)
+      .exists(_.equalsIgnoreCase("true"))
     if (!codegenFlagOn) return
     val hasExpectError = file.records.exists {
       case SqlQuery(_, _: ExpectError, _) => true
@@ -132,9 +177,7 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
   }
 
   private def runTestFile(relativePath: String, file: SqlTestFile): Unit = {
-    requireSentinelForCodegenExpectError(relativePath, file)
-    val allConfigs = ansiDisabled ++ file.configs ++ constantFoldingExcluded
-    withSQLConf(allConfigs: _*) {
+    withFixtureConfigs(file) {
       withTable(file.tables: _*) {
         file.records.foreach {
           case SqlStatement(sql, line) =>
@@ -167,6 +210,8 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
                   case Ignore(reason) =>
                     logInfo(s"IGNORED query ($reason): $sql")
                   case ExpectError(pattern) =>
+                    // Check the effective setting after any mid-file SET or RESET.
+                    requireSentinelForCodegenExpectError(relativePath, file)
                     val (sparkError, cometError) = checkSparkAnswerMaybeThrows(spark.sql(sql))
                     assert(
                       sparkError.isDefined,
@@ -192,39 +237,198 @@ class CometSqlFileTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     }
   }
 
+  test("SQL fixture optimizer directives control the optimized expression") {
+    val default = SqlFileTestParser.parse(Seq.empty)
+    withFixtureConfigs(default) {
+      val plan = spark.sql("SELECT array(1, 2)").queryExecution.optimizedPlan
+      assert(plan.expressions.exists(_.exists(_.isInstanceOf[CreateArray])))
+      val nullPlan = spark.sql("SELECT abs(CAST(NULL AS INT))").queryExecution.optimizedPlan
+      assert(!nullPlan.expressions.exists(_.exists(_.isInstanceOf[Abs])))
+    }
+    val folded = SqlFileTestParser.parse(Seq("-- ConstantFolding: enabled"))
+    withFixtureConfigs(folded) {
+      val plan = spark.sql("SELECT array(1, 2)").queryExecution.optimizedPlan
+      assert(plan.expressions.exists(_.exists {
+        case Literal(_, _: ArrayType) => true
+        case _ => false
+      }))
+      assert(!plan.expressions.exists(_.exists(_.isInstanceOf[CreateArray])))
+    }
+    val excluded = SqlFileTestParser.parse(
+      Seq("-- ExcludeRules: org.apache.spark.sql.catalyst.optimizer.NullPropagation"))
+    withFixtureConfigs(excluded) {
+      val plan = spark.sql("SELECT abs(CAST(NULL AS INT))").queryExecution.optimizedPlan
+      assert(plan.expressions.exists(_.exists(_.isInstanceOf[Abs])))
+      assert(spark.conf.get(SQLConf.OPTIMIZER_EXCLUDED_RULES.key).contains(constantFoldingRule))
+    }
+  }
+
+  test("SQL fixture optimizer Config and matrix settings retain precedence") {
+    val optimizerKey = SQLConf.OPTIMIZER_EXCLUDED_RULES.key
+    val nullPropagation = "org.apache.spark.sql.catalyst.optimizer.NullPropagation"
+    val eliminateSorts = "org.apache.spark.sql.catalyst.optimizer.EliminateSorts"
+    val file = SqlFileTestParser.parse(
+      Seq(
+        s"-- Config: $optimizerKey=$nullPropagation",
+        s"-- ConfigMatrix: $optimizerKey=$eliminateSorts,$nullPropagation",
+        s"-- ExcludeRules: $nullPropagation, $nullPropagation"))
+    val original = spark.sessionState.conf.getAllConfs
+    configMatrix(file.configMatrix).foreach { matrix =>
+      withFixtureConfigs(file.copy(configs = file.configs ++ matrix)) {
+        val selected = matrix.head._2
+        assert(
+          spark.conf.get(optimizerKey).split(",").toSeq ==
+            Seq(selected, nullPropagation, constantFoldingRule).distinct)
+      }
+      assert(spark.sessionState.conf.getAllConfs == original)
+    }
+  }
+
+  test("SQL fixture optimizer directives reject ineffective exclusions") {
+    Seq(
+      "org.apache.spark.sql.catalyst.optimizer.NoSuchRule" -> "Unknown optimizer rule",
+      spark.sessionState.optimizer.nonExcludableRules.head -> "Cannot exclude optimizer rule")
+      .foreach { case (rule, message) =>
+        val file = SqlFileTestParser.parse(Seq(s"-- ExcludeRules: $rule"))
+        assert(
+          intercept[IllegalArgumentException](withFixtureConfigs(file) {}).getMessage
+            .contains(message))
+      }
+    val conflict = SqlFileTestParser.parse(
+      Seq("-- ConstantFolding: enabled", s"-- ExcludeRules: $constantFoldingRule"))
+    assert(
+      intercept[IllegalArgumentException](withFixtureConfigs(conflict) {}).getMessage
+        .contains("conflicts"))
+  }
+
+  test("SQL fixture SET and RESET restore existing and previously unset keys") {
+    val key = "spark.comet.sqlFixture.isolation"
+    val original = spark.sessionState.conf.getAllConfs
+    assert(!original.contains(key))
+    val file = SqlFileTestParser.parse(Seq("statement", s"SET $key=first"))
+    runTestFile("set-isolation", file)
+    assert(spark.sessionState.conf.getAllConfs == original)
+    withSQLConf(key -> "original") {
+      val before = spark.sessionState.conf.getAllConfs
+      runTestFile("set-existing", file)
+      assert(spark.sessionState.conf.getAllConfs == before)
+      runTestFile("reset-existing", SqlFileTestParser.parse(Seq("statement", s"RESET $key")))
+      assert(spark.sessionState.conf.getAllConfs == before)
+      runTestFile("reset-all", SqlFileTestParser.parse(Seq("statement", "RESET")))
+      assert(spark.sessionState.conf.getAllConfs == before)
+    }
+  }
+
+  test("SQL fixture settings are restored after a failing statement or query") {
+    val key = "spark.comet.sqlFixture.isolation"
+    Seq("statement", "query").foreach { directive =>
+      val original = spark.sessionState.conf.getAllConfs
+      val file = SqlFileTestParser.parse(
+        Seq(
+          "statement",
+          s"SET $key=changed",
+          "",
+          directive,
+          "SELECT * FROM missing_sql_fixture_isolation_table"))
+      intercept[RuntimeException](runTestFile("failure-isolation", file))
+      assert(spark.sessionState.conf.getAllConfs == original)
+    }
+  }
+
+  test("SQL fixture SET changes take effect between queries") {
+    val file = SqlFileTestParser.parse(
+      Seq(
+        "statement",
+        "CREATE TABLE sql_fixture_set(v STRING) USING parquet",
+        "",
+        "statement",
+        "INSERT INTO sql_fixture_set VALUES ('invalid'), ('1')",
+        "",
+        "query",
+        "SELECT CAST(v AS INT) FROM sql_fixture_set",
+        "",
+        "statement",
+        "SET spark.sql.ansi.enabled=true",
+        "",
+        "query expect_error(CAST_INVALID_INPUT)",
+        "SELECT CAST(v AS INT) FROM sql_fixture_set"))
+    val original = spark.sessionState.conf.getAllConfs
+    runTestFile("set-between-queries", file)
+    assert(spark.sessionState.conf.getAllConfs == original)
+  }
+
+  test("SQL fixture SET cannot bypass the dispatcher error sentinel") {
+    val file = SqlFileTestParser.parse(
+      Seq(
+        "-- Config: spark.sql.ansi.enabled=true",
+        s"-- Config: ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=false",
+        "statement",
+        s"SET ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=true",
+        "",
+        "query expect_error(CAST_INVALID_INPUT)",
+        "SELECT CAST('invalid' AS INT)"))
+    val original = spark.sessionState.conf.getAllConfs
+    val error = intercept[Exception](runTestFile("set-dispatcher-sentinel", file))
+    assert(error.getMessage.contains("missing a non-error"))
+    assert(spark.sessionState.conf.getAllConfs == original)
+  }
+
+  test("SQL fixture dispatcher sentinel follows SET before an error query") {
+    val file = SqlFileTestParser.parse(
+      Seq(
+        "-- Config: spark.sql.ansi.enabled=true",
+        s"-- Config: ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=true",
+        "statement",
+        s"SET ${CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED.key}=false",
+        "",
+        "query expect_error(CAST_INVALID_INPUT)",
+        "SELECT CAST('invalid' AS INT)"))
+    val original = spark.sessionState.conf.getAllConfs
+    runTestFile("set-disable-dispatcher", file)
+    assert(spark.sessionState.conf.getAllConfs == original)
+  }
+
   // Discover and register all .sql test files
   discoverTestFiles(testResourceDir).foreach { file =>
     val relativePath = testResourceDir.toURI.relativize(file.toURI).getPath
-    val parsed = SqlFileTestParser.parse(file)
-    val combinations = configMatrix(parsed.configMatrix)
-
-    // Skip tests that fall outside the file's declared Spark version range.
-    val skip = skipReason(parsed)
-
-    if (combinations.size <= 1) {
-      // No matrix or single combination
-      test(s"sql-file: $relativePath") {
-        skip match {
-          case Some(reason) =>
-            logInfo(s"SKIPPED ($reason): $relativePath")
-          case None =>
-            val effectiveConfigs = parsed.configs ++ combinations.headOption.getOrElse(Seq.empty)
-            runTestFile(relativePath, parsed.copy(configs = effectiveConfigs))
-        }
-      }
-    } else {
-      // Multiple combinations: generate one test per combination
-      combinations.foreach { matrixConfigs =>
-        val label = matrixConfigs.map { case (k, v) => s"$k=$v" }.mkString(", ")
-        test(s"sql-file: $relativePath [$label]") {
-          skip match {
-            case Some(reason) =>
-              logInfo(s"SKIPPED ($reason): $relativePath")
-            case None =>
-              runTestFile(relativePath, parsed.copy(configs = parsed.configs ++ matrixConfigs))
+    Try(SqlFileTestParser.parse(file)) match {
+      case Failure(error) =>
+        // A malformed file must not prevent every other fixture from being registered.
+        test(s"sql-file: $relativePath") { throw error }
+      case Success(parsed) =>
+        val combinations = configMatrix(parsed.configMatrix)
+
+        // Skip tests that fall outside the file's declared Spark version range.
+        val skip = skipReason(parsed)
+
+        if (combinations.size <= 1) {
+          // No matrix or single combination
+          test(s"sql-file: $relativePath") {
+            skip match {
+              case Some(reason) =>
+                logInfo(s"SKIPPED ($reason): $relativePath")
+              case None =>
+                val effectiveConfigs =
+                  parsed.configs ++ combinations.headOption.getOrElse(Seq.empty)
+                runTestFile(relativePath, parsed.copy(configs = effectiveConfigs))
+            }
+          }
+        } else {
+          // Multiple combinations: generate one test per combination
+          combinations.foreach { matrixConfigs =>
+            val label = matrixConfigs.map { case (k, v) => s"$k=$v" }.mkString(", ")
+            test(s"sql-file: $relativePath [$label]") {
+              skip match {
+                case Some(reason) =>
+                  logInfo(s"SKIPPED ($reason): $relativePath")
+                case None =>
+                  runTestFile(
+                    relativePath,
+                    parsed.copy(configs = parsed.configs ++ matrixConfigs))
+              }
+            }
           }
         }
-      }
     }
   }
 }
diff --git a/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala b/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala
index 2b7c573bb1..f7e653c2e1 100644
--- a/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala
+++ b/spark/src/test/scala/org/apache/comet/SqlFileTestParser.scala
@@ -104,6 +104,10 @@ case class ExpectError(pattern: String) extends QueryAssertionMode
  *   Optional maximum Spark version this test applies to (e.g. "3.4"). The test is skipped on
  *   newer versions. Useful for paired fixtures where each version range has its own expected
  *   error class or output format.
+ * @param excludedRules
+ *   Additional optimizer rule names to exclude, alongside the default ConstantFolding exclusion.
+ * @param constantFoldingEnabled
+ *   Whether to let Spark fold constants before Comet sees the query.
  */
 case class SqlTestFile(
     configs: Seq[(String, String)],
@@ -111,12 +115,18 @@ case class SqlTestFile(
     records: Seq[SqlTestRecord],
     tables: Seq[String],
     minSparkVersion: Option[String] = None,
-    maxSparkVersion: Option[String] = None)
+    maxSparkVersion: Option[String] = None,
+    excludedRules: Seq[String] = Seq.empty,
+    constantFoldingEnabled: Boolean = false)
 
 object SqlFileTestParser {
 
-  private val ConfigPattern = """--\s*Config:\s*(.+)=(.+)""".r
-  private val ConfigMatrixPattern = """--\s*ConfigMatrix:\s*(.+)=(.+)""".r
+  private val ConfigPattern = """--\s*Config:\s*([^=]+)=(.*)""".r
+  private val ConfigMatrixPattern = """--\s*ConfigMatrix:\s*([^=]+)=(.*)""".r
+  private val ExcludeRulesPattern = """--\s*ExcludeRules:\s*(.*)""".r
+  private val ConstantFoldingPattern = """--\s*ConstantFolding:\s*(.*)""".r
+  private val ConfigDirectivePattern =
+    """--\s*(Config|ConfigMatrix|ExcludeRules|ConstantFolding):.*""".r
   private val MinSparkVersionPattern = """--\s*MinSparkVersion:\s*(.+)""".r
   private val MaxSparkVersionPattern = """--\s*MaxSparkVersion:\s*(.+)""".r
   private val CreateTablePattern = """(?i)CREATE\s+TABLE\s+(\w+)""".r.unanchored
@@ -125,6 +135,9 @@ object SqlFileTestParser {
     val source = Source.fromFile(file, "UTF-8")
     try {
       parse(source.getLines().toSeq)
+    } catch {
+      case e: IllegalArgumentException =>
+        throw new IllegalArgumentException(s"${file.getPath}: ${e.getMessage}", e)
     } finally {
       source.close()
     }
@@ -135,6 +148,8 @@ object SqlFileTestParser {
     var configMatrix = Seq.empty[(String, Seq[String])]
     var minSparkVersion: Option[String] = None
     var maxSparkVersion: Option[String] = None
+    var excludedRules = Seq.empty[String]
+    var constantFoldingEnabled: Option[Boolean] = None
     val records = Seq.newBuilder[SqlTestRecord]
     val tables = Seq.newBuilder[String]
 
@@ -144,13 +159,38 @@ object SqlFileTestParser {
 
       line match {
         case ConfigPattern(key, value) =>
+          require(key.trim.nonEmpty, s"Empty Config key at line ${lineIdx + 1}")
           configs :+= (key.trim -> value.trim)
           lineIdx += 1
 
         case ConfigMatrixPattern(key, values) =>
-          configMatrix :+= (key.trim -> values.split(",").map(_.trim).toSeq)
+          require(key.trim.nonEmpty, s"Empty ConfigMatrix key at line ${lineIdx + 1}")
+          val choices = values.split(",", -1).map(_.trim).toSeq
+          require(choices.forall(_.nonEmpty), s"Empty ConfigMatrix value at line ${lineIdx + 1}")
+          configMatrix :+= (key.trim -> choices)
           lineIdx += 1
 
+        case ExcludeRulesPattern(rules) =>
+          val names = splitNames(rules)
+          require(names.nonEmpty, s"Empty ExcludeRules at line ${lineIdx + 1}")
+          excludedRules ++= names
+          lineIdx += 1
+
+        case ConstantFoldingPattern(mode) =>
+          require(
+            mode == "enabled" || mode == "disabled",
+            s"Invalid ConstantFolding mode '$mode' at line ${lineIdx + 1}: " +
+              "expected enabled or disabled")
+          val enabled = mode == "enabled"
+          require(
+            constantFoldingEnabled.forall(_ == enabled),
+            s"Conflicting ConstantFolding directives at line ${lineIdx + 1}")
+          constantFoldingEnabled = Some(enabled)
+          lineIdx += 1
+
+        case ConfigDirectivePattern(name) =>
+          throw new IllegalArgumentException(s"Malformed $name directive at line ${lineIdx + 1}")
+
         case MinSparkVersionPattern(version) =>
           minSparkVersion = Some(version.trim)
           lineIdx += 1
@@ -188,7 +228,9 @@ object SqlFileTestParser {
       records.result(),
       tables.result(),
       minSparkVersion,
-      maxSparkVersion)
+      maxSparkVersion,
+      excludedRules.distinct,
+      constantFoldingEnabled.getOrElse(false))
   }
 
   private val FallbackPattern = """query\s+expect_fallback\((.+)\)""".r
diff --git a/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala b/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala
index c356c3c37a..0d107395ba 100644
--- a/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/SqlFileTestParserSuite.scala
@@ -20,6 +20,8 @@
 package org.apache.comet
 
 import java.io.File
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
 
 import org.scalatest.funsuite.AnyFunSuite
 
@@ -78,6 +80,83 @@ class SqlFileTestParserSuite extends AnyFunSuite {
       }
   }
 
+  test("configuration values preserve equals signs and empty Config values") {
+    val parsed = SqlFileTestParser.parse(
+      Seq(
+        "-- Config: fixture.key = MDEyMzQ1Njc4OTAxMjM0NQ==",
+        "-- Config: fixture.other = a=b=c",
+        "-- Config: fixture.empty=",
+        "-- ConfigMatrix: fixture.matrix = first==, second=part="))
+    assert(
+      parsed.configs == Seq(
+        "fixture.key" -> "MDEyMzQ1Njc4OTAxMjM0NQ==",
+        "fixture.other" -> "a=b=c",
+        "fixture.empty" -> ""))
+    assert(parsed.configMatrix == Seq("fixture.matrix" -> Seq("first==", "second=part=")))
+  }
+
+  test("optimizer directives preserve defaults and combine repeated exclusions") {
+    val defaults = SqlFileTestParser.parse(Seq.empty)
+    assert(!defaults.constantFoldingEnabled)
+    assert(defaults.excludedRules.isEmpty)
+    val parsed = SqlFileTestParser.parse(
+      Seq(
+        "-- ConstantFolding: enabled",
+        "-- ExcludeRules: first.Rule, second.Rule,",
+        "-- ExcludeRules: second.Rule, third.Rule"))
+    assert(parsed.constantFoldingEnabled)
+    assert(parsed.excludedRules == Seq("first.Rule", "second.Rule", "third.Rule"))
+    assert(!SqlFileTestParser.parse(Seq("-- ConstantFolding: disabled")).constantFoldingEnabled)
+  }
+
+  test("malformed configuration directives fail with their source line") {
+    Seq(
+      "-- Config: no_assignment",
+      "-- Config: =value",
+      "-- ConfigMatrix: no_assignment",
+      "-- ConfigMatrix: key=",
+      "-- ConfigMatrix: key=first,,second",
+      "-- ConfigMatrix: key=first,",
+      "-- ExcludeRules:",
+      "-- ExcludeRules: , ,",
+      "-- ConstantFolding: true").foreach { directive =>
+      val error = intercept[IllegalArgumentException] {
+        SqlFileTestParser.parse(Seq("-- comment", directive))
+      }
+      assert(error.getMessage.contains("line 2"), directive)
+    }
+    // Prose mentioning a directive is still an ordinary comment unless it includes the colon.
+    assert(
+      SqlFileTestParser.parse(Seq("-- ConstantFolding, so literals remain visible")) ==
+        SqlTestFile(Seq.empty, Seq.empty, Seq.empty, Seq.empty))
+  }
+
+  test("repeated ConstantFolding directives must agree") {
+    Seq("enabled", "disabled").foreach { mode =>
+      val directive = s"-- ConstantFolding: $mode"
+      val parsed = SqlFileTestParser.parse(Seq(directive, directive))
+      assert(parsed.constantFoldingEnabled == (mode == "enabled"))
+      val other = if (mode == "enabled") "disabled" else "enabled"
+      val error = intercept[IllegalArgumentException] {
+        SqlFileTestParser.parse(Seq(directive, s"-- ConstantFolding: $other"))
+      }
+      assert(error.getMessage.contains("Conflicting ConstantFolding directives at line 2"))
+    }
+  }
+
+  test("malformed fixture files report their path and source line") {
+    val tempDir = Files.createDirectories(new File(System.getProperty("java.io.tmpdir")).toPath)
+    val path = Files.createTempFile(tempDir, "comet-malformed-fixture-", ".sql")
+    try {
+      Files.write(path, "-- comment\n-- Config: no_assignment".getBytes(StandardCharsets.UTF_8))
+      val error = intercept[IllegalArgumentException](SqlFileTestParser.parse(path.toFile))
+      assert(error.getMessage.contains(path.toString))
+      assert(error.getMessage.contains("line 2"))
+    } finally {
+      Files.delete(path)
+    }
+  }
+
   test("statements and queries preserve SQL, source lines and tables for cleanup") {
     val lines = Seq(
       "-- fixture comment",


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org