You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "namanjain24-sudo (via GitHub)" <gi...@apache.org> on 2026/09/11 14:28:46 UTC
[PR] chore(ci): Register the substrait check job with runs-on, route spark checks through xtask [datafusion]
namanjain24-sudo opened a new pull request, #25195:
URL: https://github.com/apache/datafusion/pull/25195
## Which issue does this PR close?
- Part of #25148. Also finishes the `cargo check` half of #21048, which asks for the CI steps to be reproducible locally.
## Rationale for this change
While measuring the `cargo check <crate> features` jobs for #25148 I found two inconsistencies in them. Neither is a policy question about which runner to use, so they seemed worth separating out.
**1. The substrait check job never registers with `runs-on`.**
Fifteen jobs in `rust.yml` select a `runs-on` runner with `extras=s3-cache`. Fourteen of them start with the `runs-on/action` step that registers the job. `cargo check datafusion-substrait features` is the one that does not, so on `push` and `merge_group` it lands on the `runs-on` runner but gets neither the external cache nor the job metrics the other fourteen report. It looks like an oversight rather than a choice; the job directly above and below it in the file both have the step.
**2. The spark check job is the last one not going through `xtask`.**
`cargo check datafusion-spark features` calls `cargo check` directly. Every other check step in the workflow goes through `cargo xtask ci step check`, which is what lets a contributor run the same thing locally. After this PR there is no raw `cargo check` left in `rust.yml`.
## What changes are included in this PR?
Four lines.
- Add the `runs-on/action` step to `linux-datafusion-substrait-features`, matching its siblings.
- Replace the three `datafusion-spark` `cargo check` invocations with the `xtask` equivalents.
The commands `xtask` emits are the same ones being replaced, which `--explain` confirms:
| step | before | `cargo xtask ci step check datafusion-spark …` |
| --- | --- | --- |
| default features | `cargo check --profile ci --all-targets -p datafusion-spark` | `default` |
| no-default-features | `cargo check --profile ci --no-default-features -p datafusion-spark` | `no-default` |
| core | `cargo check --profile ci --no-default-features -p datafusion-spark --features=core` | `core` |
I deliberately left out the larger changes those jobs invite, because they are the open question in #25148 rather than something to decide in a cleanup PR. For the record, what I measured there: five of the seven `cargo check <crate> features` jobs declare no `Rust Dependency Cache` step at all, only `datafusion` and `datafusion-substrait` do; three of the seven are pinned to `ubuntu-latest` rather than the `vars.USE_RUNS_ON` expression; and across those seven jobs about a third of the wall clock is setup and cache restore paid seven times over. Happy to follow up on any of that if it is wanted.
## What is the testing strategy for this PR?
The three `xtask` commands were run locally against this branch and all pass:
```
cargo xtask ci step check datafusion-spark default
cargo xtask ci step check datafusion-spark no-default
cargo xtask ci step check datafusion-spark core
```
`python3 ci/scripts/check_asf_yaml_status_checks.py` reports `OK: All 32 required_status_checks match existing GitHub Actions jobs` — job names are unchanged, so the required checks are unaffected. `ci/scripts/check_no_cargo_install_in_workflows.sh` and `ci/scripts/typos_check.sh` are clean, and the file still parses as YAML with the same 26 jobs.
The `runs-on/action` half cannot be verified outside the Apache repo, since the action is a no-op on standard GitHub runners.
## Are there any user-facing changes?
No. CI configuration only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore(ci): Register the substrait check job with runs-on, route spark checks through xtask [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #25195:
URL: https://github.com/apache/datafusion/pull/25195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore(ci): Register the substrait check job with runs-on, route spark checks through xtask [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #25195:
URL: https://github.com/apache/datafusion/pull/25195#issuecomment-5791699971
Hi @namanjain24-sudo Thank you for your work. I see you have 13 PRs open at the moment https://github.com/apache/datafusion/pull/25645
These kind of volume create burden for the maintainers who will have to review and merge all the prs. I suggest you pick three of the prs which you think are most important at the moment and close others for now. As the prs get merged you can reopen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore(ci): Register the substrait check job with runs-on, route spark checks through xtask [datafusion]
Posted by "namanjain24-sudo (via GitHub)" <gi...@apache.org>.
namanjain24-sudo commented on PR #25195:
URL: https://github.com/apache/datafusion/pull/25195#issuecomment-5803003154
Fair point, thanks for saying it directly @kumarUjjawal. Most of these are small, independent substrait fixes rather than one big thing split up, so they don't block each other — but I get that review load doesn't care about that.
A few are already approved and just waiting on merge (#25146, #25498, #25529). I'll hold off on new ones for now, and happy to close whichever ones you'd deprioritize — just point me at them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore(ci): Register the substrait check job with runs-on, route spark checks through xtask [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25195:
URL: https://github.com/apache/datafusion/pull/25195#issuecomment-5724676896
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25195?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 82.34%. Comparing base ([`edc936f`](https://app.codecov.io/gh/apache/datafusion/commit/edc936f38b749dec58eacdc1d64a28e2854c55cd?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`03065ec`](https://app.codecov.io/gh/apache/datafusion/commit/03065ec3bc2fa233985bb0245a5cb39952d97d33?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25195 +/- ##
========================================
Coverage 82.33% 82.34%
========================================
Files 1137 1137
Lines 432498 432716 +218
Branches 432498 432716 +218
========================================
+ Hits 356116 356334 +218
- Misses 54843 54847 +4
+ Partials 21539 21535 -4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25195?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore(ci): Register the substrait check job with runs-on, route spark checks through xtask [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #25195:
URL: https://github.com/apache/datafusion/pull/25195#issuecomment-5830935435
> @kumarUjjawal
>
> Did not mean to overrule you.
>
> This was a small PR to review.
@kosiew Completely fine. I was going to review it was just an initial passing remark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore(ci): Register the substrait check job with runs-on, route spark checks through xtask [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #25195:
URL: https://github.com/apache/datafusion/pull/25195#issuecomment-5830862851
@kumarUjjawal
Did not mean to overrule you.
This was a small PR to review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org