You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/20 08:42:20 UTC

[PR] test: let Spark 4 test profiles use the Spark-default ANSI mode [datafusion-comet]

andygrove opened a new pull request, #4370:
URL: https://github.com/apache/datafusion-comet/pull/4370

   ## Which issue does this PR close?
   
   Closes #.
   
   Builds on the idea behind #4368.
   
   ## Rationale for this change
   
   `CometTestBase` was pinning `spark.sql.ansi.enabled` to `false`, so the Spark 4.0/4.1/4.2 test profiles never actually exercised the ANSI behavior real Spark 4 users get out of the box. Tests that pass against Comet today may regress in production because we never run them with the same defaults.
   
   ## What changes are included in this PR?
   
   - Stop setting `ANSI_ENABLED` in `CometTestBase.sparkConf`. Spark 3.x picks up its own default (off), Spark 4.x picks up its own default (on).
   - For suites whose tests predominantly exercise silent-overflow / wrap-around / null-on-bad-input semantics, override `sparkConf` at the suite level to disable ANSI (tests that target ANSI behavior already opt in via `withSQLConf(ANSI_ENABLED -> "true")`):
     - `CometFuzzMathSuite`
     - `CometCastSuite`
     - `CometAggregateSuite`
   - `CometSqlFileTestSuite`: prepend `ANSI_ENABLED -> "false"` to the conf list passed into `withSQLConf`. The `_ansi.sql` fixtures that include `--CONFIG: spark.sql.ansi.enabled=true` still win, because their values are applied later.
   - Per-test `withSQLConf(SQLConf.ANSI_ENABLED.key -> "false")` wraps for the long tail in `CometExpressionSuite`, `CometArrayExpressionSuite`, `CometMathExpressionSuite`, `CometDateTimeUtilsSuite`, `CometExecSuite`, and `ParquetReadSuite`.
   
   ## How are these changes tested?
   
   The existing tests are the test. Ran the affected suites under `-Pspark-4.0` locally (892 tests across 10 suites, 0 failures):
   
   | Suite | Tests | Result |
   |---|---|---|
   | `CometFuzzMathSuite` | 30 | pass |
   | `CometCastSuite` | 165 | pass |
   | `CometAggregateSuite` | 81 | pass |
   | `CometExpressionSuite` | 127 | pass |
   | `CometArrayExpressionSuite` | 39 | pass |
   | `CometMathExpressionSuite` | 7 | pass |
   | `CometDateTimeUtilsSuite` | 8 | pass |
   | `CometExecSuite` | 124 | pass |
   | `CometSqlFileTestSuite` | 258 | pass |
   | `ParquetReadV1Suite` | 53 | pass |
   
   CI will sweep Spark 4.0/4.1/4.2 across all the partitions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] test: let Spark 4 test profiles use the Spark-default ANSI mode [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #4370:
URL: https://github.com/apache/datafusion-comet/pull/4370#issuecomment-4499083047

   > LGTM, do we need a plan to enhance the ANSI-specific test paths, or do we think those are well exercised by the existing tests that came in with ANSI-specific PRs?
   > 
   > Thanks @andygrove!
   
   I think we're good. We have explicit ANSI opt-in testing for all supported expressions that have different behavior for ANSI mode. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] test: let Spark 4 test profiles use the Spark-default ANSI mode [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4370:
URL: https://github.com/apache/datafusion-comet/pull/4370


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org