You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "blaginin (via GitHub)" <gi...@apache.org> on 2026/02/02 11:14:02 UTC
[PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
blaginin opened a new pull request, #20107:
URL: https://github.com/apache/datafusion/pull/20107
## Which issue does this PR close?
Related to https://github.com/apache/datafusion/issues/13813
Thanks to the infra team and @gmcdonald specifically, we now have the ability to use more powerful AWS-provided runners in our CI 🥳
DataFusion has one of the largest runtimes across Apache projects - that's why we're bringing those runners here first. Since we're first to test this, I think it's reasonable to do a gradual transition, so I updated the two most frequently failing actions to be hosted in AWS. The plan is to test that everything works fine and then transition the remaining actions.
## What changes are included in this PR?
If the org is `apache`, we'll now use ASF-provisioned runners in the ASF infra AWS account. Forks will not have access to those runners, so they will fall back to GitHub-provisioned ones.
## Are these changes tested?
Yes.
- Forks in non-apache orgs: https://github.com/blaginin/datafusion/pull/10
- Apache org: https://github.com/apache/datafusion-sandbox/pull/163
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3837444901
I think it is a great idea to try one or two jobs initially and then slowly migrate over
The biggest concern I have is if there is some difference between the normal runners (that will run on PRs) and the one that will run on main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #20107:
URL: https://github.com/apache/datafusion/pull/20107
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3850017360
Yes, it is around twice faster now for the affected CI jobs!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3850435009
Thanks @blaginin I think we need the same for Comet, as its heavily tested. Appreciate if you can share how to start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3860965002
FYI, noticed that for some reason when you rerun the action, it doesn't get picked up by the new runners 🤔
One important implication of that is that people who need their workflow to be approved won't be able to run the CI (since the first run is auto-cancelled as unapproved).
ASF Infra team will have a debugging session with the Runs-on team Monday evening, and I also submitted a GitHub ticket. In the meanwhile, if that's blocking anyone, feel free to merge this https://github.com/apache/datafusion/pull/20170 and I'll roll it back hopefully on Monday 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3938914584
Follow up switching more actions: https://github.com/apache/datafusion/pull/20274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3876741954
Found the issue! We had a very low on-demand quota on the account. We were bumping the spot quota, but reruns use on-demand, which is why they weren't managing to start. Submitted a request - hopefully everything should work fine today, and I'll switch over more workers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3849967199
BTW I don't have data, but this feels like it made a major difference on CI time recently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3837357117
> Are you aware if any limits applies for AWS runners, I checked https://docs.github.com/en/actions/reference/limits but cannot say if it is for AWS runners
Yes, those limits do not apply to us. We will be bounded by the ASF AWS Account budget and service quotas, this will be on me and infra team to monitor
I will merge this PR soon and we can start testing 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on code in PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#discussion_r2763544648
##########
.github/workflows/rust.yml:
##########
@@ -45,10 +45,11 @@ jobs:
# Check crate compiles and base cargo check passes
linux-build-lib:
name: linux build test
- runs-on: ubuntu-latest
+ runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m7a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }}
Review Comment:
> `github.repository_owner == 'apache'`
❤️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3854408320
> > Appreciate if you can share how to start
>
> Of course! I'll switch the remaining DF actions and then happy to do the same with Comet. I'll also write a guide for transitioning :)
thanks and FYI created https://github.com/apache/datafusion-comet/issues/3404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3834657648
Nice !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3852619690
> Appreciate if you can share how to start
Of course! I'll switch the remaining DF actions and then happy to do the same with Comet. I'll also write a guide for transitioning :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3877348136
https://github.com/apache/datafusion/actions/runs/21847381185/job/63104381656
reruns are working now 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3838457188
If everyone is okay, I'm merging the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Runs-on for `linux-build-lib` and `linux-test` (2X faster CI) [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #20107:
URL: https://github.com/apache/datafusion/pull/20107#issuecomment-3840407458
> The biggest concern I have is if there is some difference between the normal runners (that will run on PRs) and the one that will run on main
Those on `main` will also use AWS runners, here's an [example](https://github.com/apache/datafusion/actions/runs/21624028030/job/62319851901#step:18:17) 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org