You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by yh...@apache.org on 2024/01/30 18:03:45 UTC

(beam) branch master updated: Fix Dataproc cleanup race condition (#30154)

This is an automated email from the ASF dual-hosted git repository.

yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
     new b9fd39cb16b Fix Dataproc cleanup race condition (#30154)
b9fd39cb16b is described below

commit b9fd39cb16b1900120ea1d4b1b2844c3468321d0
Author: Yi Hu <ya...@google.com>
AuthorDate: Tue Jan 30 13:03:37 2024 -0500

    Fix Dataproc cleanup race condition (#30154)
---
 .github/workflows/README.md                         | 1 +
 .github/workflows/beam_CleanUpDataprocResources.yml | 2 +-
 .test-infra/dataproc/cleanup.sh                     | 4 ++--
 3 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/README.md b/.github/workflows/README.md
index b39dfc9d592..f882553ef9b 100644
--- a/.github/workflows/README.md
+++ b/.github/workflows/README.md
@@ -79,6 +79,7 @@ jobs:
       github.event_name == 'push' ||
       github.event_name == 'pull_request_target' ||
       (github.event_name == 'schedule' && github.repository == 'apache/beam') ||
+      github.event_name == 'workflow_dispatch' ||
       startsWith(github.event.comment.body, 'Run Job With Matrix')
     steps:
       - uses: actions/checkout@v3
diff --git a/.github/workflows/beam_CleanUpDataprocResources.yml b/.github/workflows/beam_CleanUpDataprocResources.yml
index 86a3e42c389..9e56230ffdb 100644
--- a/.github/workflows/beam_CleanUpDataprocResources.yml
+++ b/.github/workflows/beam_CleanUpDataprocResources.yml
@@ -17,7 +17,7 @@ name: Cleanup Dataproc Resources
 
 on:
   schedule:
-    - cron: '0 */6 * * *'
+    - cron: '0 0 * * *'
   workflow_dispatch:
 
 #Setting explicit permissions for the action to avoid the default permissions which are `write-all` in case of pull_request_target event
diff --git a/.test-infra/dataproc/cleanup.sh b/.test-infra/dataproc/cleanup.sh
index ac12a0bff05..95eea525a2e 100755
--- a/.test-infra/dataproc/cleanup.sh
+++ b/.test-infra/dataproc/cleanup.sh
@@ -44,8 +44,8 @@ function filterClusters(){
 
 		elapsedHours=$((($currentDate - $clusterStartTime)/3600))
 
-		#Most of the jobs commonly finish in less than one hour
-		if [[ $elapsedHours -ge 2 ]]; then 
+		# teardown clusters >= 6 hr old
+		if [[ $elapsedHours -ge 6 ]]; then
 			for name in ${generatedResources[@]}; do
 				# Only resources generated by the groovy jobs set are queued for deletion
 				if [[ "$cluster" == *${name}* && ! ("$cluster" =~ nokill) ]]; then