You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/13 13:00:24 UTC

[GitHub] [airflow] potiuk edited a comment on issue #11496: Concurrency limit for CI jobs

potiuk edited a comment on issue #11496:
URL: https://github.com/apache/airflow/issues/11496#issuecomment-707721268


   Seems also that splitting to smaller jobs has this side effect that it makes competing with other projects much harder. If there are many projects that are holding their slots for a long time, we can get  starved rather heavily ( I think this is what is happening now). I am thinking about combining the jobs back, but in a smart way. 
   
   Instead of having full set of tests as matrix strategy to run them sequentially in one job and perform clean-up after each "type" of tests.  This way we can avoid the Pylint memory leaks and in-container memory leaks. I can draft it quickly.
   
   We can clean-up file-system and run `docker system prune` between each type of tests.  I think that would be the best of both worlds.
   
   WDYT @kaxil @ashb @dimberman  ?
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org