You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kiley Sok (Jira)" <ji...@apache.org> on 2022/03/24 20:04:00 UTC
[jira] [Updated] (BEAM-12903) Implement a solution to cleanup GCP resources in beam testing projects
[ https://issues.apache.org/jira/browse/BEAM-12903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kiley Sok updated BEAM-12903:
-----------------------------
Description:
FR to create a solution to clean up lingering GCE, VMs, Dataproc clusters, etc.
Some sort of perodic job which runs and deletes leaked GCE VMs would prevent the resource concerns caused by failed tests as in:
https://issues.apache.org/jira/browse/BEAM-12898
Possible solutions:
* Name all the resources with a convention that represents a TTL. Then the resources can be deleted by a cron/periodic job if the TTL has expired
* Dataproc cluster may have a TTL configuration option itself which the flink on dataproc test can use.
* Similar to how we delete old Dataflow jobs in https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
was:
FR to create a solution to clean up lingering GCE, VMs, Dataproc clusters, etc.
Some sort of perodic job which runs and deletes leaked GCE VMs would prevent the resource concerns caused by failed tests as in:
https://issues.apache.org/jira/browse/BEAM-12898?filter=-2
Possible solutions:
* Name all the resources with a convention that represents a TTL. Then the resources can be deleted by a cron/periodic job if the TTL has expired
* Dataproc cluster may have a TTL configuration option itself which the flink on dataproc test can use.
> Implement a solution to cleanup GCP resources in beam testing projects
> ----------------------------------------------------------------------
>
> Key: BEAM-12903
> URL: https://issues.apache.org/jira/browse/BEAM-12903
> Project: Beam
> Issue Type: Test
> Components: testing
> Reporter: Alex Amato
> Priority: P3
>
> FR to create a solution to clean up lingering GCE, VMs, Dataproc clusters, etc.
> Some sort of perodic job which runs and deletes leaked GCE VMs would prevent the resource concerns caused by failed tests as in:
> https://issues.apache.org/jira/browse/BEAM-12898
> Possible solutions:
> * Name all the resources with a convention that represents a TTL. Then the resources can be deleted by a cron/periodic job if the TTL has expired
> * Dataproc cluster may have a TTL configuration option itself which the flink on dataproc test can use.
> * Similar to how we delete old Dataflow jobs in https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_CancelStaleDataflowJobs.groovy
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)