You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/03 12:56:18 UTC
[GitHub] [beam] Abacn opened a new issue, #23020: [Bug]: Kubernetes service not cleaned up and reaching quota causing multiple performance test failure
Abacn opened a new issue, #23020:
URL: https://github.com/apache/beam/issues/23020
### What happened?
Multiple performance tests perma-red since Aug 31st, including JDBC, Kafka, IO Tests on HDFS. The cause is the number of existing k8s endpoints has run out of quota. These end points all have namespace beam-performancetests-[testname]-[test-number]. From the test number it can be seen there are all old tests. Usually k8s service gets deleted at the end of the test. But if jenkins worker fails in the middle, which happened on Aug 30, those service is never cleaned up.
Another leak found is that rc validation script creates a k8s cluster named after "xlang-kafka" for kafka taxi test:
https://github.com/apache/beam/blob/31561e2ff13147aa80f9f811e2a94ebe57b25374/release/src/main/scripts/run_rc_validation.sh#L501
, but never gets deleted. Consequently every time rc validation there is a new cluster created and at some point it will reach cluster quota.
There should be a cron job automatically clean up these services and clusters.
### Issue Priority
Priority: 1
### Issue Component
Component: test-failures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn closed issue #23020: [Bug]: Kubernetes service not cleaned up and reaching quota causing multiple performance test failure
Posted by GitBox <gi...@apache.org>.
tvalentyn closed issue #23020: [Bug]: Kubernetes service not cleaned up and reaching quota causing multiple performance test failure
URL: https://github.com/apache/beam/issues/23020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org