You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/06 22:52:41 UTC
[GitHub] [airflow] potiuk opened a new pull request, #26904: Add resource debugging to all parallel runs
potiuk opened a new pull request, #26904:
URL: https://github.com/apache/airflow/pull/26904
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26904: Fix running parallel k8s tests in "depth-first" mode in CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26904:
URL: https://github.com/apache/airflow/pull/26904#issuecomment-1272968452
One last rebase to check with "full tests"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26904: Fix running parallel k8s tests in "depth-first" mode in CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26904:
URL: https://github.com/apache/airflow/pull/26904#issuecomment-1272789169
Hey here - this is something I had to fix before I disconnect for the next 2 weeks.
It looks I managed to solve some of tha main (not all but should be pretty good) stability issues with "K8S tasks". They are now consistently failing in main when run in parallel and I mitigated the problem by:
* decreasing paralllelism (accounting by the fact that Kind cluster + airflow require many more resources than 1 CPU only.
* using `depth-first` approach for running the tests (not all k8s clusters need to be started upfront when run in parallel - they are now created and discarded in parallel as needed and no more than 2 clusters are running in parallel ).
* fixing some parallelism-incompatible bugs
* improving diagnostics to be much more usable in case of failures (already used it to fix some of the bugs).
I tested it with self-hosted and public runners and while could not eliminate every single flakiness (there are still few tests that randomly fail) - it is much closer to way better stability and much easier to diagnose problem and locally reproduce it (@ephraimbuddy - you might be glad to hear it, I think the previous CI logs were next to unusable).
I would love to merge it soon - this will help to keep main `green` (and this is the last one that will keep me from completely disconnecting so I'd appreciate quick turnaround :).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26904: Add resource debugging to all parallel runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26904:
URL: https://github.com/apache/airflow/pull/26904#issuecomment-1270762611
I added debugging resources in all places in order to debug K8S tests failure in main. Turned out that it was running out of space in /var/lib/docker filesystem, so I increased the space we allocate for `/var/lib/docker` in our AMI image. Hopefully it will run fine (but the main should be fixed already without merging this one - this one only enables debugging the resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #26904: Add resource debugging to all parallel runs
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #26904: Add resource debugging to all parallel runs
URL: https://github.com/apache/airflow/pull/26904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #26904: Add resource debugging to all parallel runs
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #26904: Add resource debugging to all parallel runs
URL: https://github.com/apache/airflow/pull/26904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #26904: Fix running parallel k8s tests in "depth-first" mode in CI
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #26904:
URL: https://github.com/apache/airflow/pull/26904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26904: Add resource debugging to all parallel runs
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26904:
URL: https://github.com/apache/airflow/pull/26904#issuecomment-1270797426
It was a bit too small so I got it bumped a little again. 🤞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #26904: Add resource debugging to all parallel runs
Posted by GitBox <gi...@apache.org>.
potiuk closed pull request #26904: Add resource debugging to all parallel runs
URL: https://github.com/apache/airflow/pull/26904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26904: Fix running parallel k8s tests in "depth-first" mode in CI
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26904:
URL: https://github.com/apache/airflow/pull/26904#issuecomment-1273431168
All green. Looking forward to merge this one :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org