You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2026/03/20 08:02:20 UTC
[PR] [GLUTEN-11550][UT] Replace GlutenTestsCommonTrait with correct Gluten test traits for Spark 4.0/4.1 [gluten]
baibaichen opened a new pull request, #11800:
URL: https://github.com/apache/gluten/pull/11800
### What changes were proposed in this pull request?
`GlutenTestsCommonTrait` does **not** configure `GlutenPlugin` or create a Gluten-enabled `SparkSession`. Test suites using this trait were running on vanilla Spark, falsely reporting "offload to gluten".
This PR replaces `GlutenTestsCommonTrait` with the correct traits in 157 test files (spark40 + spark41):
| Category | Count | Target Trait | Effect |
|----------|:-----:|-------------|--------|
| SQL tests (QueryTest/SharedSparkSession) | 105 | `GlutenSQLTestsTrait` / `GlutenSQLTestsBaseTrait` | SparkSession with GlutenPlugin via `sparkConf` override |
| Expression + Unit tests | 73 | `GlutenTestsTrait` (spark41: `shim.GlutenTestsTrait`) | SparkSession with GlutenPlugin via `initializeSession()` |
| Hive tests (TestHiveSingleton) | 7 | Kept as `GlutenTestsCommonTrait` | TestHive singleton cannot inject GlutenPlugin |
| BroadcastJoinSuite | 1 | Kept as `GlutenTestsCommonTrait` | Already self-creates GlutenPlugin SparkSession |
47 suites (spark41) / 42 suites (spark40) are disabled with `// TODO: 4.x` markers after failing under the correct traits. 25 of these are newly discovered failures; the rest were already tracked in #11550.
> [#11512](https://github.com/apache/gluten/pull/11512) introduced PlanStability test suites for Spark 4.0 and 4.1 by extending Spark's original suites with `GlutenTestsCommonTrait`. While the tests appeared to pass, they were **not actually loading the Gluten plugin** — they were effectively running vanilla Spark, which trivially passes golden file comparison against Spark's own approved plans.
### How was this patch tested?
- Compiled both spark40 and spark41 (`test-compile` with mvnd)
- Ran all 140+ enabled suites on spark41 via `run-scala-test.sh --mvnd` — all passed
- Ran all 119+ enabled suites on spark40 — all passed
- Ran the 40 spark41-excluded suites on spark40 to identify version-specific differences (5 suites pass on spark40 but fail on spark41)
### New disabled suites discovered by this PR
These suites previously appeared to pass because they were not loading GlutenPlugin:
#### catalyst.expressions
| Suite | Spark 4.0 | Spark 4.1 |
|-------|-----------|-----------|
| `GlutenCastWithAnsiOnSuite` | 🔴 (4 failures) | 🔴 (10 failures) |
| `GlutenCollationRegexpExpressionsSuite` | 🟢 | 🔴 (1 failure) |
| `GlutenExpressionEvalHelperSuite` | 🔴 (2 failures) | 🔴 (2 failures) |
| `GlutenObjectExpressionsSuite` | 🔴 (7 failures) | 🔴 (7 failures) |
| `GlutenOrderingSuite` | 🟢 | 🔴 (2 failures) |
| `GlutenScalaUDFSuite` | 🔴 (1 failure) | 🔴 (1 failure) |
| `GlutenToPrettyStringSuite` | 🔴 (1 failure) | 🔴 (1 failure) |
#### execution
| Suite | Spark 4.0 | Spark 4.1 |
|-------|-----------|-----------|
| `GlutenCoalesceShufflePartitionsSuite` | 🔴 (ABORTED) | 🔴 (ABORTED) |
| `GlutenColumnarRulesSuite` | 🟢 | 🔴 (1 failure) |
| `GlutenDataSourceScanExecRedactionSuite` | 🔴 (2 failures) | 🔴 (2 failures) |
| `GlutenDataSourceV2ScanExecRedactionSuite` | 🔴 (2 failures) | 🔴 (2 failures) |
| `GlutenExternalAppendOnlyUnsafeRowArraySuite` | 🔴 (14 failures) | 🔴 (14 failures) |
| `GlutenHiveResultSuite` | 🟢 | 🔴 (1 failure) |
| `GlutenLogicalPlanTagInSparkPlanSuite` | 🔴 (ABORTED) | 🔴 (ABORTED) |
| `GlutenProjectedOrderingAndPartitioningSuite` | 🔴 (6 failures) | 🔴 (6 failures) |
| `GlutenRemoveRedundantProjectsSuite` | 🔴 (14 failures) | 🔴 (14 failures) |
| `GlutenSQLExecutionSuite` | 🔴 (1 failure) | 🔴 (1 failure) |
| `GlutenSQLJsonProtocolSuite` | 🔴 (1 failure) | 🔴 (1 failure) |
| `GlutenShufflePartitionsUtilSuite` | 🔴 (1 failure) | 🔴 (1 failure) |
| `GlutenSimpleSQLViewSuite` | 🔴 (1 failure) | 🔴 (2 failures) |
| `GlutenUnsafeRowSerializerSuite` | 🔴 (1 failure) | 🔴 (1 failure) |
#### execution.datasources.parquet
| Suite | Spark 4.0 | Spark 4.1 |
|-------|-----------|-----------|
| `GlutenParquetCommitterSuite` | 🔴 (spark.test.home) | 🔴 (spark.test.home) |
#### org.apache.spark.sql
| Suite | Spark 4.0 | Spark 4.1 |
|-------|-----------|-----------|
| `GlutenJoinHintSuite` | 🔴 (1 failure) | 🔴 (1 failure) |
| `GlutenRandomDataGeneratorSuite` | 🟢 | 🔴 (232 failures) |
#### streaming
| Suite | Spark 4.0 | Spark 4.1 |
|-------|-----------|-----------|
| `GlutenTransformWithStateClusterSuite` | 🔴 (spark.test.home) | 🔴 (spark.test.home) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][UT] Replace GlutenTestsCommonTrait with correct Gluten test traits for Spark 4.0/4.1 [gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen merged PR #11800:
URL: https://github.com/apache/gluten/pull/11800
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][UT] Replace GlutenTestsCommonTrait with correct Gluten test traits for Spark 4.0/4.1 [gluten]
Posted by "weiting-chen (via GitHub)" <gi...@apache.org>.
weiting-chen commented on code in PR #11800:
URL: https://github.com/apache/gluten/pull/11800#discussion_r2970964797
##########
gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala:
##########
@@ -694,42 +694,42 @@ class VeloxTestSettings extends BackendTestSettings {
enableSuite[GlutenAggregatingAccumulatorSuite]
enableSuite[GlutenCoGroupedIteratorSuite]
enableSuite[GlutenColumnarRulesSuite]
- enableSuite[GlutenDataSourceScanExecRedactionSuite]
- enableSuite[GlutenDataSourceV2ScanExecRedactionSuite]
+ // TODO: 4.x enableSuite[GlutenDataSourceScanExecRedactionSuite] // 2 failures
+ // TODO: 4.x enableSuite[GlutenDataSourceV2ScanExecRedactionSuite] // 2 failures
enableSuite[GlutenExecuteImmediateEndToEndSuite]
- enableSuite[GlutenExternalAppendOnlyUnsafeRowArraySuite]
+ // TODO: 4.x enableSuite[GlutenExternalAppendOnlyUnsafeRowArraySuite] // 14 failures
enableSuite[GlutenGlobalTempViewSuite]
enableSuite[GlutenGlobalTempViewTestSuite]
enableSuite[GlutenGroupedIteratorSuite]
enableSuite[GlutenHiveResultSuite]
// TODO: 4.x enableSuite[GlutenInsertSortForLimitAndOffsetSuite] // 6 failures
enableSuite[GlutenLocalTempViewTestSuite]
- enableSuite[GlutenLogicalPlanTagInSparkPlanSuite]
+ // TODO: 4.x enableSuite[GlutenLogicalPlanTagInSparkPlanSuite] // RUN ABORTED
enableSuite[GlutenOptimizeMetadataOnlyQuerySuite]
enableSuite[GlutenPersistedViewTestSuite]
// TODO: 4.x enableSuite[GlutenPlannerSuite] // 1 failure
- enableSuite[GlutenProjectedOrderingAndPartitioningSuite]
+ // TODO: 4.x enableSuite[GlutenProjectedOrderingAndPartitioningSuite] // 6 failures
enableSuite[GlutenQueryPlanningTrackerEndToEndSuite]
- enableSuite[GlutenRemoveRedundantProjectsSuite]
+ // TODO: 4.x enableSuite[GlutenRemoveRedundantProjectsSuite] // 14 failures
// TODO: 4.x enableSuite[GlutenRemoveRedundantSortsSuite] // 1 failure
enableSuite[GlutenRowToColumnConverterSuite]
- enableSuite[GlutenSQLExecutionSuite]
+ // TODO: 4.x enableSuite[GlutenSQLExecutionSuite] // 1 failure
enableSuite[GlutenSQLFunctionSuite]
- enableSuite[GlutenSQLJsonProtocolSuite]
- enableSuite[GlutenShufflePartitionsUtilSuite]
- enableSuite[GlutenSimpleSQLViewSuite]
+ // TODO: 4.x enableSuite[GlutenSQLJsonProtocolSuite] // 1 failure
+ // TODO: 4.x enableSuite[GlutenShufflePartitionsUtilSuite] // 1 failure
+ // TODO: 4.x enableSuite[GlutenSimpleSQLViewSuite] // 1 failure
// TODO: 4.x enableSuite[GlutenSparkPlanSuite] // 1 failure
enableSuite[GlutenSparkPlannerSuite]
enableSuite[GlutenSparkScriptTransformationSuite]
enableSuite[GlutenSparkSqlParserSuite]
enableSuite[GlutenUnsafeFixedWidthAggregationMapSuite]
enableSuite[GlutenUnsafeKVExternalSorterSuite]
- enableSuite[GlutenUnsafeRowSerializerSuite]
+ // TODO: 4.x enableSuite[GlutenUnsafeRowSerializerSuite] // 1 failure
// TODO: 4.x enableSuite[GlutenWholeStageCodegenSparkSubmitSuite] // 1 failure
// TODO: 4.x enableSuite[GlutenWholeStageCodegenSuite] // 24 failures
enableSuite[GlutenBroadcastExchangeSuite]
enableSuite[GlutenLocalBroadcastExchangeSuite]
- enableSuite[GlutenCoalesceShufflePartitionsSuite]
+ // TODO: 4.x enableSuite[GlutenCoalesceShufflePartitionsSuite] // LiveListenerBus stopped
// Rewrite for Gluten. Change details are in the inline comments in individual tests.
.excludeByPrefix("determining the number of reducers")
Review Comment:
Should this line be commented out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-11550][UT] Replace GlutenTestsCommonTrait with correct Gluten test traits for Spark 4.0/4.1 [gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on code in PR #11800:
URL: https://github.com/apache/gluten/pull/11800#discussion_r2971114921
##########
gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala:
##########
@@ -694,42 +694,42 @@ class VeloxTestSettings extends BackendTestSettings {
enableSuite[GlutenAggregatingAccumulatorSuite]
enableSuite[GlutenCoGroupedIteratorSuite]
enableSuite[GlutenColumnarRulesSuite]
- enableSuite[GlutenDataSourceScanExecRedactionSuite]
- enableSuite[GlutenDataSourceV2ScanExecRedactionSuite]
+ // TODO: 4.x enableSuite[GlutenDataSourceScanExecRedactionSuite] // 2 failures
+ // TODO: 4.x enableSuite[GlutenDataSourceV2ScanExecRedactionSuite] // 2 failures
enableSuite[GlutenExecuteImmediateEndToEndSuite]
- enableSuite[GlutenExternalAppendOnlyUnsafeRowArraySuite]
+ // TODO: 4.x enableSuite[GlutenExternalAppendOnlyUnsafeRowArraySuite] // 14 failures
enableSuite[GlutenGlobalTempViewSuite]
enableSuite[GlutenGlobalTempViewTestSuite]
enableSuite[GlutenGroupedIteratorSuite]
enableSuite[GlutenHiveResultSuite]
// TODO: 4.x enableSuite[GlutenInsertSortForLimitAndOffsetSuite] // 6 failures
enableSuite[GlutenLocalTempViewTestSuite]
- enableSuite[GlutenLogicalPlanTagInSparkPlanSuite]
+ // TODO: 4.x enableSuite[GlutenLogicalPlanTagInSparkPlanSuite] // RUN ABORTED
enableSuite[GlutenOptimizeMetadataOnlyQuerySuite]
enableSuite[GlutenPersistedViewTestSuite]
// TODO: 4.x enableSuite[GlutenPlannerSuite] // 1 failure
- enableSuite[GlutenProjectedOrderingAndPartitioningSuite]
+ // TODO: 4.x enableSuite[GlutenProjectedOrderingAndPartitioningSuite] // 6 failures
enableSuite[GlutenQueryPlanningTrackerEndToEndSuite]
- enableSuite[GlutenRemoveRedundantProjectsSuite]
+ // TODO: 4.x enableSuite[GlutenRemoveRedundantProjectsSuite] // 14 failures
// TODO: 4.x enableSuite[GlutenRemoveRedundantSortsSuite] // 1 failure
enableSuite[GlutenRowToColumnConverterSuite]
- enableSuite[GlutenSQLExecutionSuite]
+ // TODO: 4.x enableSuite[GlutenSQLExecutionSuite] // 1 failure
enableSuite[GlutenSQLFunctionSuite]
- enableSuite[GlutenSQLJsonProtocolSuite]
- enableSuite[GlutenShufflePartitionsUtilSuite]
- enableSuite[GlutenSimpleSQLViewSuite]
+ // TODO: 4.x enableSuite[GlutenSQLJsonProtocolSuite] // 1 failure
+ // TODO: 4.x enableSuite[GlutenShufflePartitionsUtilSuite] // 1 failure
+ // TODO: 4.x enableSuite[GlutenSimpleSQLViewSuite] // 1 failure
// TODO: 4.x enableSuite[GlutenSparkPlanSuite] // 1 failure
enableSuite[GlutenSparkPlannerSuite]
enableSuite[GlutenSparkScriptTransformationSuite]
enableSuite[GlutenSparkSqlParserSuite]
enableSuite[GlutenUnsafeFixedWidthAggregationMapSuite]
enableSuite[GlutenUnsafeKVExternalSorterSuite]
- enableSuite[GlutenUnsafeRowSerializerSuite]
+ // TODO: 4.x enableSuite[GlutenUnsafeRowSerializerSuite] // 1 failure
// TODO: 4.x enableSuite[GlutenWholeStageCodegenSparkSubmitSuite] // 1 failure
// TODO: 4.x enableSuite[GlutenWholeStageCodegenSuite] // 24 failures
enableSuite[GlutenBroadcastExchangeSuite]
enableSuite[GlutenLocalBroadcastExchangeSuite]
- enableSuite[GlutenCoalesceShufflePartitionsSuite]
+ // TODO: 4.x enableSuite[GlutenCoalesceShufflePartitionsSuite] // LiveListenerBus stopped
// Rewrite for Gluten. Change details are in the inline comments in individual tests.
.excludeByPrefix("determining the number of reducers")
Review Comment:
cool!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org