[GitHub] [airflow] potiuk commented on pull request #26904: Fix running parallel k8s tests in "depth-first" mode in CI

[GitBox <gi...@apache.org>]

potiuk commented on PR #26904:
URL: https://github.com/apache/airflow/pull/26904#issuecomment-1272789169

   Hey here - this is something I had to fix before I disconnect for the next 2 weeks.
   
   It looks I managed to solve some of tha main (not all but should be pretty good) stability issues with "K8S tasks". They are now consistently failing in main when run in parallel and I mitigated the problem by:
   
   * decreasing paralllelism (accounting by the fact that Kind cluster + airflow require many more resources than 1 CPU only.
   * using `depth-first` approach for running the tests (not all k8s clusters need to be started upfront when run in parallel - they are now created and discarded in parallel as needed and no more than 2 clusters are running in parallel ).
   * fixing some parallelism-incompatible bugs
   * improving diagnostics to be much more usable in case of failures (already used it to fix some of the bugs).
   
   I tested it with self-hosted and public runners and while could not eliminate every single flakiness (there are still few tests that randomly fail) - it is much closer to way better stability and much easier to diagnose problem and locally reproduce it (@ephraimbuddy - you might be glad to hear it, I think the previous CI logs were next to unusable).
   
   I would love to merge it soon - this will help to keep main `green` (and this is the last one that will keep me from completely disconnecting so I'd appreciate quick turnaround :).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org