You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/25 14:13:49 UTC
[PR] ci: Consolidate Spark SQL test jobs to reduce CI time [datafusion-comet]
andygrove opened a new pull request, #3271:
URL: https://github.com/apache/datafusion-comet/pull/3271
## Summary
Merges three near-identical Spark SQL test jobs into a single consolidated job with a config matrix:
- `spark-sql-auto-scan` (was 20 jobs)
- `spark-sql-native-native-comet` (was 14 jobs)
- `spark-sql-native-iceberg-compat` (was 14 jobs, now 7)
**Before:** 48 jobs across 3 workflows (293 lines)
**After:** 41 jobs in 1 consolidated workflow (191 lines)
**Reduction:** ~15% fewer jobs
### Test matrix
| Scan Implementation | Spark Versions | Jobs |
|---------------------|----------------|------|
| auto | 3.4, 3.5, 4.0 | 20 |
| native_comet | 3.4, 3.5 | 14 |
| native_iceberg_compat | 3.5 | 7 |
The `native_iceberg_compat` scanner is now only tested on Spark 3.5, as testing on 3.4 provides diminishing returns.
## Test plan
- [ ] Verify the workflow runs correctly on this PR
- [ ] Check that all expected matrix combinations are generated
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Consolidate Spark SQL test jobs to reduce CI time [datafusion-comet]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3271:
URL: https://github.com/apache/datafusion-comet/pull/3271#issuecomment-3796739092
## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3271?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 22.06%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e6b7750`](https://app.codecov.io/gh/apache/datafusion-comet/commit/e6b775072c63debb58ecf9a0f13b6c79e34025ee?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 879 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3271 +/- ##
=============================================
- Coverage 56.12% 22.06% -34.07%
+ Complexity 976 512 -464
=============================================
Files 119 172 +53
Lines 11743 15961 +4218
Branches 2251 2641 +390
=============================================
- Hits 6591 3522 -3069
- Misses 4012 11885 +7873
+ Partials 1140 554 -586
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3271?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Consolidate Spark SQL test jobs to reduce CI time [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3271:
URL: https://github.com/apache/datafusion-comet/pull/3271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: Consolidate Spark SQL test jobs to reduce CI time [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3271:
URL: https://github.com/apache/datafusion-comet/pull/3271#issuecomment-3801929903
> Thanks @andygrove I def like the clean up, however I'm not sure how 15% fewer jobs are calculated, this PR has 118 checks completed vs #3266 (122 checks)
15% fewer Spark SQL jobs, not 15% fewer total jobs, I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org