You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/14 22:52:17 UTC

[GitHub] [airflow] potiuk opened a new pull request #11538: More stable kubernetes port forwarding

potiuk opened a new pull request #11538:
URL: https://github.com/apache/airflow/pull/11538

Seems that port forwarding during kubernetes tests started to behave
erratically - seems that kubectl port forward sometimes might hang
indefinitely rather than connect or fail.
We change the strategy a bit to try to allocate
increasing port numbers in case something like that happens.

<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.

Feel free to ping committers for the review!

In case of existing issue, reference it using one of the following:

closes: #ISSUE
related: #ISSUE

How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->

---
**^ Add meaningful description above**

Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #11538: More stable kubernetes port forwarding

Posted by GitBox <gi...@apache.org>.

potiuk merged pull request #11538:
URL: https://github.com/apache/airflow/pull/11538


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11538: More stable kubernetes port forwarding

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11538:
URL: https://github.com/apache/airflow/pull/11538#issuecomment-708990762

@kaxil @ashb @turbaszek -> this one should solve the Kubernetes problems we started to experienced recently. They apparently were not related to the provider split as I originally suspected - but to some changes in the way how port forwarding started to interact with GA runner. So looking forward to reviews :+1:

One more thing and maybe you can help me verify my theory.

I believe GA is kinda reusing workers without full restarts between them - that might be the reason for 137 errors and resource exhaustion because they do not clean up the machines fully.

It could be an accident this is the only explanation for an error I saw yesterday that some other jobs were affected by the kubectl background processes that we started in other jobs. This was an earlier version of the fix, but it did not have the trap that kills (first gently and then forcefully) all kubectl instances running in the background:

https://github.com/apache/airflow/runs/1256383093?check_suite_focus=true

There were seemingly unrelated errors (in several other jobs). Seems like for other jobs (theoretically in different machines!), the tests were affected by the background-running hanging kubectls, as if the 8080 port numbers continued to be be "taken". I am not 100% sure of that, but that is the only explanation I have for this. The errors went completely away when I added the trap to kill the kubectls (in unrelated jobs !).

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #11538: More stable kubernetes port forwarding

Posted by GitBox <gi...@apache.org>.

potiuk edited a comment on pull request #11538:
URL: https://github.com/apache/airflow/pull/11538#issuecomment-708739987


   Hey @dimberman -> I hope this one will fix the recent problems with kubernetes port-forward stability. There were a lot of prroblems caused by it recently and I implemented another workaround - increasing port numbers between subsequent tries. I thin ti will solve the problem for good.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #11538: More stable kubernetes port forwarding

Posted by GitBox <gi...@apache.org>.

potiuk commented on pull request #11538:
URL: https://github.com/apache/airflow/pull/11538#issuecomment-708739987


   Hey @dimberman -> I hope this one will fix the recent problems with kubernetes port-forward stability. There were a lot of prroblems caused by it recently and I implemented another workaround - incerasing port numbers between subsequent tries. I thin ti will solve the problem for good.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org