You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2025/05/20 15:38:53 UTC
[I] The Clean Up GCP Resources job is flaky [beam]
github-actions[bot] opened a new issue, #31846:
URL: https://github.com/apache/beam/issues/31846
The Clean Up GCP Resources is failing over 80% of the time.
Please visit https://github.com/apache/beam/actions/workflows/beam_CleanUpGCPResources.yml?query=is%3Afailure+branch%3Amaster to see all failed workflow runs.
See also Grafana statistics: http://metrics.beam.apache.org/d/CTYdoxP4z/ga-post-commits-status?orgId=1&viewPanel=1&var-Workflow=Clean%20Up%20GCP%20Resources
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "akashorabek (via GitHub)" <gi...@apache.org>.
akashorabek closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2303620917
due to 2.58.1 validation Python PostCommits run again without #31955 and leaked dataset
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2760102392
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2580688196
It has been a recurring issue: #33435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2609264059
Job has been stable for 2 weeks. Closing as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2610197147
Amar3tto@ Thanks for keeping an eye for this. This is mitigated as we skipped clean up k8s workload that has non-empty finalizer for now (#33545). However we still need to fix the clean up script so it works for such workflow (otherwise stale workload will still accumulate as time goes)
Re-opened #33435 for tracking this follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2566829881
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2942646701
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2764389278
Job has been stable for 3 days. Closing as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-4237826038
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2956874518
Maybe we can display Failed deletes as a warning instead of an error? Someone with permissions could manually delete them by ID.
Or it would be better to find the tests that produce such datasets and clean them up after the test. @Abacn What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2303608032
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2922688398
Job has been stable for 1 day. Closing as resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2556222332
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2575870692
Cleaned up some stale resources, got a green run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2894925727
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "akashorabek (via GitHub)" <gi...@apache.org>.
akashorabek commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2739351212
There were couple of datasets which were probably created by tests ran with different service account. I granted the main service account the permission to delete them and it fixed the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2579122584
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2223311574
there are more "Tried and failed to delete bq_read_all_" and other datasets. Likely related to recent python test workflow failure, which used a different service account to run tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2735242240
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2919786022
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "akashorabek (via GitHub)" <gi...@apache.org>.
akashorabek closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #31846:
URL: https://github.com/apache/beam/issues/31846#issuecomment-2956949798
> Maybe we can display Failed deletes as a warning instead of an error?
I'm fine to move it to warning.
Checked again, the dataset is created at test pipeline execution time on worker, so it is owned by GCE compute service account, and github action service account doesn't have permission to delete it
The code points to here: https://github.com/apache/beam/blob/df54bf439b28b23c3380b913811633fd8678c403/sdks/python/apache_beam/io/gcp/bigquery_tools.py#L351C3-L351C15 so any pipeline created a temp dataset (usually using FILE_LOAD that generated multiple load jobs) could have this leftovers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The Clean Up GCP Resources job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #31846: The Clean Up GCP Resources job is flaky
URL: https://github.com/apache/beam/issues/31846
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org