You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/23 14:56:55 UTC
[PR] ci: scope Spark SQL trigger paths to per-version shims and diff [datafusion-comet]
andygrove opened a new pull request, #4415:
URL: https://github.com/apache/datafusion-comet/pull/4415
## Summary
Tighten the path allow-list on each `spark_sql_test_<v>.yml` so a change confined to one Spark version no longer fans out to the others.
Each workflow currently triggers on `spark/src/main/**` and `dev/diffs/**`. Today that means:
- editing `spark/src/main/spark-4.1/` runs the 3.5 and 4.0 workflows for no reason
- editing `dev/diffs/3.4.3.diff` runs the 4.0 workflow for no reason
- editing `dev/diffs/iceberg/*.diff` runs all four Spark SQL workflows for no reason
- if we ever add a `spark/src/main/spark-4.3/` shim it will run the 3.x workflows for no reason
The change is purely to the `paths:` filters; job logic is untouched.
### Per-version mapping (from `pom.xml`)
| Profile | Shim dirs that apply | Diff |
|---|---|---|
| spark-3.4 | `spark-3.4`, `spark-3.x` | `dev/diffs/3.4.3.diff` |
| spark-3.5 | `spark-3.5`, `spark-3.x` | `dev/diffs/3.5.8.diff` |
| spark-4.0 | `spark-4.0`, `spark-4.x` | `dev/diffs/4.0.2.diff` |
| spark-4.1 | `spark-4.1`, `spark-4.x` | `dev/diffs/4.1.1.diff` |
Each workflow keeps `spark/src/main/**` (so unrelated Java/Scala/resources still trigger) but adds `!`-exclusions for the shim dirs that don't apply, and replaces `dev/diffs/**` with the single applicable diff file.
Both `push:` and `pull_request:` filters are updated where present (3.4 and 4.1 only have `push:` paths since their PR trigger is `labeled`).
## Test plan
- [ ] actionlint passes on all four workflow files
- [ ] After merge, edit only `spark/src/main/spark-4.1/...` in a follow-up PR and confirm only the 4.1 workflow is queued
- [ ] After merge, edit only `dev/diffs/3.5.8.diff` and confirm only the 3.5 workflow is queued
- [ ] After merge, edit only `dev/diffs/iceberg/1.10.0.diff` and confirm none of the Spark SQL workflows are queued
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] ci: scope Spark SQL trigger paths to per-version shims and diff [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #4415:
URL: https://github.com/apache/datafusion-comet/pull/4415
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org