You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/13 12:02:58 UTC
[GitHub] [airflow] potiuk opened a new issue #11496: Concurrency limit for CI jobs
potiuk opened a new issue #11496:
URL: https://github.com/apache/airflow/issues/11496
Yesterday's problem with CI builds has been confirmed as hitting concurrency limit by GitHub support.
According to this page [Limits](https://docs.github.com/en/free-pro-team@latest/actions/reference/usage-limits-billing-and-administration#usage-limits) we have 180 concurrent job limits at the Apache Enterprise-level account. Seems that we are competing with other projects (similarly as it was in Travis).
I started a thread on the builds@apache.org list https://lists.apache.org/thread.html/r1708881f52adbdae722afb8fea16b23325b739b254b60890e72375e1%40%3Cbuilds.apache.org%3E
and opened an accompanying ticket in JIRA: https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20978
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #11496:
URL: https://github.com/apache/airflow/issues/11496
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707721268
Seems also that splitting to smaller jobs has this side effect that it makes competing with other projects much harder. If there are many projects that are holding their slots for a long time, we can get starved rather heavily ( I think this is what is happening now). I am thinking about combining the jobs back, but in a smart way.
Instead of having full set of tests as matrix strategy to run them sequentially in one job and perform clean-up after each "type" of tests. This way we can avoid the Pylint memory leaks and in-container memory leaks. I can draft it quickly.
We can clean-up file-system and run `docker system prune` between each type of tests. I think that would be the best of both worlds.
WDYT @kaxil @ashb @dimberman ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707694164
@aijamalnk Re: self-hosted runners -> Would it be possible to come back about the discussion on Google Donating some credits to us ? The previous ones expired, and we did not use them to the full extent as we were not ready, but right now with the current GitHub Actions full support for running self-hosted runners in Google Cloud https://github.blog/2020-08-04-github-actions-self-hosted-runners-on-google-cloud/ I am sure we can start using those immediately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707692599
cc: @kaxil @ashb @dimberman @mik-laj @turbaszek (and others).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707774002
@potiuk Sounds good! lets try it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707721268
Seems also that splitting to smaller jobs has this side effect that it makes competing with other projects much harder. If there are many projects that are holding their slots for a long time, we can get starved rather heavily ( I think this is what is happening now). I am thinking about combining the jobs back, but in a smart way.
Instead of having full set of tests as matrix strategy, I think we can run them sequentially in one job and perform clean-up after each "type" of tests. This way we can avoid the Pylint memory leaks and in-container memory leaks. I can draft it quickly after we get 2.0alpha out.
We can clean-up file-system and run `docker system prune` between each type of tests. I think that would be the best of both worlds.
WDYT @kaxil @ashb @dimberman ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707777493
Yes please, +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-750360985
Closed - already superseded
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707829808
Got it working - also combined the jobs for kubernetes the same way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707721268
Seems also that splitting to smaller jobs has this side effect that it makes competing with other projects much harder. If there are many projects that are holding their slots for a long time, we can get starved rather heavily ( I think this is what is happening now). I am thinking about combining the jobs back, but in a smart way.
Instead of having full set of tests as matrix strategy, I think we can run them sequentially in one job and perform clean-up after each "type" of tests. This way we can avoid the Pylint memory leaks and in-container memory leaks. I can draft it quickly.
We can clean-up file-system and run `docker system prune` between each type of tests. I think that would be the best of both worlds.
WDYT @kaxil @ashb @dimberman ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11496: Concurrency limit for CI jobs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707721268
Seems also that splitting to smaller jobs has this side effect that it makes competing with other projects much harder. If there are many projects that are holding the slot for a long time, we can get starved rather heavily ( I think this is what is happening now). I am thinking about combining the jobs back, but in a smart way.
Instead of having full set of tests as matrix strategy to run them sequentially in one job and perform clean-up after each "type" of tests. This way we can avoid the Pylint memory leaks and in-container memory leaks. I can draft it quickly.
We can clean-up file-system and run `docker system prune` between each type of tests. I think that would be the best of both worlds.
WDYT @kaxil @ashb @dimberman ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org