You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/09 22:44:51 UTC
[GitHub] [airflow] khalidmammadov opened a new pull request #19499: Align runner selection rule to ci.yml
khalidmammadov opened a new pull request #19499:
URL: https://github.com/apache/airflow/pull/19499
This is to align runner selection rules to ci.yml so builds run on hosted runners for non contributors rather than on self-hosted.
Non contributors dont have access to self-hosted runners and cant influence them by any means (i.e. restart, increase etc.)
This will make "build-image" job to run ubuntu and independent from self-hosted runner.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-964619644
Why would that be better? What do you want to achieve this way?
No-one can restart any runners - this is How GitHub Actions work.
Having build images run on self-hosted runners was a deliberate decision, so I wonder why you'd like to change it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-965146267
Yeah. It's double-edged sword. Often self-hosted are usually faster than the public runners.
However @ashb preparing to the build meeting tomorrow, I was looking at the latest charts which I added here: https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-11 it looks like GitHub Actions implemented some more "fair" algorithm of distributing public runners to the projects in the meantime.
Seems that there are no mores peaks of ~500 queued jobs from one project (which was the main reason our 150 job queue was stuck for literally hours. And I have not seen many CI builds/PRs from users waiting for a looong time. What I saw is what @khalidmammadov observed - that because we have limited queue of self-hosted, the regular PRs timed out because their `build-image` workflow did not have a chance to complete due to many committers pushing their PRs.
I think maybe it actually makes sense to bring the build-image jobs from regular PRs to use public runners ? At lest to try. In VAST majority of cases those are 1 - 2 minute jobs and all of them are non-parallel (docker builds do not parallelise well) so they won't be much slower.
WDYT @ashb ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-964978837
The reason we use the self hosted runners is that the queue on them is _shorter_ than using the hosted ones, cos as a project we share a pool with _all_ projects in the Apache org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-965154410
See the charts - even if spark is using more and more GH , it does not seem to have significant impact on other projects, It seems that it mostly impacts spark's queue numbers - but not the other's queue numbers - that tells me that the algorithm might be much more "fair" now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-965146267
Yeah. It's double-edged sword. Often self-hosted are usually faster than the public runners.
However @ashb preparing to the buidl meeting tomorrow, I was looking at the latest charts which I added here: https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-11 it looks like GitHub Actions implemented some more "fair" algorithm of distributing public runners to the projects in the meantime.
Seems that there are no mores peaks of ~500 queued jobs from one project (which was the main reason our 150 job queue was stuck for literally hours. And I have not seen many CI builds/PRs from users waiting for a looong time. What I saw is what @khalidmammadov observed - that because we have limited queue of self-hosted, the regular PRs timed out because their `build-image` workflow did not have a chance to complete due to many committers pushing their PRs.
I think maybe it actually makes sense to bring the build-image jobs from regular PRs to use public runners ? At lest to try. In VAST majority of cases those are 1 - 2 minute jobs and all of them are non-parallel (docker builds do not parallelise well) so they won't be much slower.
WDYT @ashb ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-964983019
The issue is there is a problem with the self-hosted runners right now. I'll look at and fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
ashb closed pull request #19499:
URL: https://github.com/apache/airflow/pull/19499
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-964619644
Why would that be better? What do you want to achieve this way?
No-one can restart them - having build images run on self-hosted runners was a deliberate decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] khalidmammadov commented on pull request #19499: Align runner selection rule to ci.yml
Posted by GitBox <gi...@apache.org>.
khalidmammadov commented on pull request #19499:
URL: https://github.com/apache/airflow/pull/19499#issuecomment-964883252
I was working on PRs and noticed that build-images phase waits for self-hosted runner to be available. It took long time and caused checks to fail. When checked I can see other PRs are similarly queued and waiting for self-hosted. And I dont know/see where they are hosted nor see what's wrong with them and why they take some much time to get allocated. But all others that run hosted ones are completing just fine.
And looking into ci.yml I saw this diff. Hence PR.
Hope it make sence. If you think we rather need to check what's causing self-hosted ones response slowly then we can close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org