You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/18 22:24:31 UTC
[GitHub] [airflow] potiuk opened a new pull request, #25143: Speed up Kubernetes tests 2x-3x times.
potiuk opened a new pull request, #25143:
URL: https://github.com/apache/airflow/pull/25143
Currently K8S tests are too limited in resources and running too many
combinations of Python/K8S versions.
Regarding the resources - we were limiting them to 1/4 of available
CPUs but monitoring shows that ~20% memory and up to 4 CPUs are used
when 2 tests are running in parallel (on a big instance).
We can safely run all 5 parallell tests at the same time with big
instances speeding up the test up to 3 times when full tests are
needed.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25143: Speed up Kubernetes tests 2x-3x times on main
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25143:
URL: https://github.com/apache/airflow/pull/25143#issuecomment-1188699093
> Looks like it was much faster, good find!
Not THAT fast :). This one only run a single combo of K8s version + Python - (I already disable multiple versions of Helm tests outside main :). I will re-push it with draft changes to actually run multiple combos.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25143: Speed up Kubernetes tests 2x-3x times on main
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25143:
URL: https://github.com/apache/airflow/pull/25143#issuecomment-1188708080
Ah... Found a typo in ci.yml :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25143: Speed up Kubernetes tests 30% on main
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25143:
URL: https://github.com/apache/airflow/pull/25143#issuecomment-1188944358
Right. We go down to 16 minutes instead of 23 on main - so more or less what expected. Only upgrade is still long - but I have an idea how to speed up this one too (PR shortly).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25143: Speed up Kubernetes tests 30% on main
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25143:
URL: https://github.com/apache/airflow/pull/25143#issuecomment-1188889085
Yeah it was a bit too "optimistic". When I run finally 5 tests in parallel (after fixing #25148 ) 2 of the 5 tests started to fail - and likely lack of memory in "peaks" was the culprit. However increasing the parallel tests execution to 3 should already help to decrease the time ~30%.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #25143: Speed up Kubernetes tests 2x-3x times.
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #25143:
URL: https://github.com/apache/airflow/pull/25143#issuecomment-1188384532
Let's see if my observations are correct - but this should speed up a number of K8S tests quite a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25143: Speed up Kubernetes tests 30% on main
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25143:
URL: https://github.com/apache/airflow/pull/25143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org