You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "pabloem (via GitHub)" <gi...@apache.org> on 2025/01/17 16:39:07 UTC
[I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
pabloem opened a new issue, #33644:
URL: https://github.com/apache/beam/issues/33644
### What needs to happen?
Many tests and examples in the Beam codebase create GCP assets that are not cleaned up. Some examples:
- BQ datasets
- Pubsub topics, subscriptions
- BigTable instances?
- Others?
The task is to build automation that can **safely** disable-then-clean-up leftover assets from the GCP environment. The tool would:
- List existing inventory of various kinds of assets
- Figure out which ones are meant to be cleaned up (i.e. short-lived test assets, vs. long-lived project assets like base sources, base buckets, etc).
- Wait a week or so
- Disable and (eventually?) delete these assets
Why?
- A lot of these assets have billing implications for the project
- A lot of these assets make navigating the project more difficult (e.g. reviewing Bigquery datasets)
Risks:
- We need to roll this out carefully to make sure we don't delete important assets. We'd probably start with strict lists of inclusions (i.e. everything that's not included will be excluded).
### Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [x] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-5034338133
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-2612908379
cc @Amar3tto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "ksobrenat32 (via GitHub)" <gi...@apache.org>.
ksobrenat32 commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-3339078488
The PubSub part was covered on #34141 , #35553 and #35548. Will check the rest :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "MansiSingh17 (via GitHub)" <gi...@apache.org>.
MansiSingh17 commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-3938348742
@pabloem Hi, I'm interested in working on this project as part of GSoC 2026.
I've been contributing to Apache Beam's TypeScript SDK — my merged PRs include LRU cache eviction for CachingStateProvider (#37214) and CombinePerKeyPrecombineOperator (#37466). I also have a background in IT audit and compliance (ISO 27001, SOC 2, GDPR) from previous roles, which I think is relevant to the IaC, drift detection, and key management aspects of this project.
A few questions to make sure I understand the current scope:
1. Is the GCP asset cleanup piece (#33644) still open, or is the focus now on IaC for access permissions and drift detection?
2. Is there a preferred IaC tool (Terraform, Pulumi, etc.) for the Beam infrastructure work?
3. Would it make sense to start with a small contribution in the infra area before drafting a proposal?
Happy to dig into the existing workflows and share ideas. Looking forward to your thoughts!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-2611006996
In Beam repo's CI/CD, there is a workflow, runs daily, to clean up the resources:
https://github.com/apache/beam/blob/master/.github/workflows/beam_CleanUpGCPResources.yml
though this does not cover the case if test is running in a fork.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-5369976500
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "pabloem (via GitHub)" <gi...@apache.org>.
pabloem commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-2598933072
Sample of this issue: https://github.com/apache/beam/issues/19760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33644:
URL: https://github.com/apache/beam/issues/33644#issuecomment-3000323207
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Task]: Build a cleaner for assets in the GCP test environment [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33644: [Task]: Build a cleaner for assets in the GCP test environment
URL: https://github.com/apache/beam/issues/33644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org