You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/09 20:35:31 UTC
[GitHub] [beam] elink21 opened a new pull request, #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
elink21 opened a new pull request, #21779:
URL: https://github.com/apache/beam/pull/21779
Addresses #21191 , a new cron job was created in order to monitor and delete leaked Dataproc clusters, only clusters that are part of the list of generated resources will be queued for deletion if they've been active for an abnormal period of time, this will avoid issues like the ones reported in https://issues.apache.org/jira/browse/BEAM-12898.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Mention the appropriate issue in your description (for example: "addresses #21191), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment "fixes #<ISSUE NUMBER>" instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #21779:
URL: https://github.com/apache/beam/pull/21779#discussion_r896909691
##########
.test-infra/dataproc/cleanup.sh:
##########
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -Eeuo pipefail
+GCLOUD_REGION="us-central1"
+clustersList=( )
+toDeleteList=( )
+
+
+generatedResources=("beam-loadtests-go-cogbk-flink" "beam-loadtests-python-cogbk-flink" \
Review Comment:
It looks like it happened in the past due to some errors in the code, the misconfiguration was fixed in [BEAM-12898](https://issues.apache.org/jira/browse/BEAM-12898), other possible scenarios might be the existence of interrupted or aborted jobs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #21779:
URL: https://github.com/apache/beam/pull/21779#issuecomment-1154530830
I should say that I am not an expert in the various unix utilities used here, so I am trusting your work and testing to confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
elink21 commented on PR #21779:
URL: https://github.com/apache/beam/pull/21779#issuecomment-1151607686
R: Hi @aaltay , Kyle had more context about this issue. Do you know if someone else can help reviewing and seeding the new job?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] elink21 commented on a diff in pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
elink21 commented on code in PR #21779:
URL: https://github.com/apache/beam/pull/21779#discussion_r897381545
##########
.test-infra/dataproc/cleanup.sh:
##########
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -Eeuo pipefail
+GCLOUD_REGION="us-central1"
+clustersList=( )
+toDeleteList=( )
+
+
+generatedResources=("beam-loadtests-go-cogbk-flink" "beam-loadtests-python-cogbk-flink" \
Review Comment:
During the testing process for [BEAM-12898](https://issues.apache.org/jira/browse/BEAM-12898) sometimes when the Jenkins node went completely down due to overloading, PostBuild Scripts were not executed, this situation is rare though, not sure if events like the Jenkins downtime that happened a few days ago would also trigger this.
The computation load of this job is not high , an option could be to execute it less frequently just as a backup in case that something eventually happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a diff in pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
y1chi commented on code in PR #21779:
URL: https://github.com/apache/beam/pull/21779#discussion_r897092923
##########
.test-infra/dataproc/cleanup.sh:
##########
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -Eeuo pipefail
+GCLOUD_REGION="us-central1"
+clustersList=( )
+toDeleteList=( )
+
+
+generatedResources=("beam-loadtests-go-cogbk-flink" "beam-loadtests-python-cogbk-flink" \
Review Comment:
The PostBuild scripts are supposed to run even if the task is failed or aborted, there was a fix previously in https://github.com/apache/beam/commit/a84d9cb285c5d15eb34f4a3126015ae1f1da003f.
I'm not against adding additional clean up script, but do have concern that it is duplicating the clean up steps and overtime this may become outdated more easily than maintaining the PostBuild steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles merged pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
kennknowles merged PR #21779:
URL: https://github.com/apache/beam/pull/21779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] y1chi commented on a diff in pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
y1chi commented on code in PR #21779:
URL: https://github.com/apache/beam/pull/21779#discussion_r896241176
##########
.test-infra/dataproc/cleanup.sh:
##########
@@ -0,0 +1,68 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -Eeuo pipefail
+GCLOUD_REGION="us-central1"
+clustersList=( )
+toDeleteList=( )
+
+
+generatedResources=("beam-loadtests-go-cogbk-flink" "beam-loadtests-python-cogbk-flink" \
Review Comment:
do we know when the cleanup https://github.com/apache/beam/blob/e27d8d6bb744daa098e21fc71e6371ca5f2062cf/.test-infra/jenkins/Flink.groovy#L96 is not working?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
kennknowles commented on PR #21779:
URL: https://github.com/apache/beam/pull/21779#issuecomment-1156886910
OK it looks good now that the question is answered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21779:
URL: https://github.com/apache/beam/pull/21779#issuecomment-1151637785
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21779:
URL: https://github.com/apache/beam/pull/21779#issuecomment-1151637786
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #21779: [BEAM-12903] Cron job to cleanup Dataproc leaked resources
Posted by GitBox <gi...@apache.org>.
aaltay commented on PR #21779:
URL: https://github.com/apache/beam/pull/21779#issuecomment-1151617912
I do not know who will have context. @kileys / @kennknowles / @y1chi - do you know who could review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org