You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "srielau (via GitHub)" <gi...@apache.org> on 2026/08/20 14:47:52 UTC
[PR] [SPARK-58909][SQL][TESTS] Move heavy suites out of the sql-other GHA shard [spark]
srielau opened a new pull request, #58169:
URL: https://github.com/apache/spark/pull/58169
### What changes were proposed in this pull request?
Rebalance the GitHub Actions `sql - other tests` shard by tagging the slowest untagged SQL suites so they run in `sql - extended tests` or `sql - slow tests`.
`@SlowSQLTest` (streaming family, same destination as SPARK-53068):
- `TransformWithStateInitialStateSuiteCheckpointV2` (parent was already tagged; this subclass was not)
- `RealTimeTransformWithStateSuite` and `RealTimeTransformWithStateSuiteWithRowChecksum`
- `StatePartitionAllColumnFamiliesWriterSuite`
- `StreamingSourceEvolutionSuite`
`@ExtendedSQLTest`:
- `AdaptivePartialAggregationSuite`, `SortSuite`
- Default collation V1/V2 STRING and CHAR/VARCHAR suites
- `CollationSuite`, `ParquetGeoSuite`, `ParquetTypeWideningSuite`
- `OrcV1QuerySuite` / `OrcV2QuerySuite`, `CSVv1Suite` / `CSVv2Suite`
- `InsertSuite` (`sql/core`), `KeyGroupedPartitioningSuite`, `SQLLastAttemptMetricPlanShapesSuite`
- `RocksDBDecoupledMaintenanceSuite` and `RocksDBDecoupledMaintenanceSuiteWithRowChecksum`
No workflow or timeout changes.
### Why are the changes needed?
`sql - other tests` is running against the 150-minute `build` job timeout. On GitHub-hosted runners it recently finished in about 141 minutes (https://github.com/srielau/spark/actions/runs/32331062736), so a slightly slower runner times the job out with no useful test report.
The other SQL shards still have spare capacity (`sql - extended tests` ~56 minutes, `sql - slow tests` ~105 minutes on the same run). Moving these suites is the same approach as SPARK-44034 / SPARK-53068 / SPARK-54460.
Suite list came from a successful `sql - other` log, using each suite's ScalaTest start banner and thread-leak end marker. The tagged set is about 35-40 minutes of wall time.
JIRA: https://issues.apache.org/jira/browse/SPARK-58909
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Test-only annotations; no production code change. Existing suites are unchanged besides the CI tag. GitHub Actions on this PR will show whether `sql - other`, `sql - slow`, and `sql - extended` all finish under the 150-minute cap.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58909][SQL][TESTS] Move heavy suites out of the sql-other GHA shard [spark]
Posted by "srielau (via GitHub)" <gi...@apache.org>.
srielau commented on PR #58169:
URL: https://github.com/apache/spark/pull/58169#issuecomment-5359793298
Thanks @gengliangwang. Done in 7e685c3:
- Left `CSVv1Suite` and `OrcV1QuerySuite` in `sql-other`.
- Kept `CSVv2Suite` and `OrcV2QuerySuite` as `ExtendedSQLTest`.
- Tagged the other file-source `*V2Suite` counterparts the same way (`JsonV2Suite`, `TextV2Suite`, `WholeTextFileV2Suite`, `OrcSourceV2Suite`, `ParquetFileFormatV2Suite`, `ParquetV2QuerySuite`, `MetadataCacheV2Suite`, `NestedDataSourceV2Suite`, `ExplodeEmbeddedArrayJsonV2Suite`, `CollatedFilterPushDownToParquetV2Suite`, `PushVariantIntoScanV2Suite`).
Left suites that were already `@SlowSQLTest` (`ParquetRebaseDatetimeV2Suite`, `FileStreamSinkV2Suite`, `StreamingDataSourceV2Suite`) on the slow shard. Did not retag API-named suites such as `DataSourceV2Suite` / `JDBCV2Suite` / `DataFrameWriterV2Suite`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58909][SQL][TESTS] Move heavy suites out of the sql-other GHA shard [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #58169: [SPARK-58909][SQL][TESTS] Move heavy suites out of the sql-other GHA shard
URL: https://github.com/apache/spark/pull/58169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58909][SQL][TESTS] Move heavy suites out of the sql-other GHA shard [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #58169:
URL: https://github.com/apache/spark/pull/58169#issuecomment-5359743386
@srielau thanks for the improvement!
Instead of putting all `OrcV1QuerySuite / OrcV2QuerySuite, CSVv1Suite / CSVv2Suite` as `ExtendedSQLTest`, shall we tag all the `*V2Suite` as `ExtendedSQLTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58909][SQL][TESTS] Move heavy suites out of the sql-other GHA shard [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #58169:
URL: https://github.com/apache/spark/pull/58169#issuecomment-5361646073
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/f7f32dbe1c78ca499c1dc2eae53f6a532f50bc3e
- merged into branch-4.x https://github.com/apache/spark/commit/1151c6fa3f1ca546f13f56befe87ebada9f643a6
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org