You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "damccorm (via GitHub)" <gi...@apache.org> on 2025/09/19 16:19:28 UTC
[I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
damccorm opened a new issue, #36214:
URL: https://github.com/apache/beam/issues/36214
### What would you like to happen?
Add a transform which allows users to use a secret manager to ensure that any GBK data is encrypted before being sent to a runner. This allows users to avoid unencrypted data at rest.
Subparts:
- [ ] Add GBEK transform in Python
- [ ] Add GBEK transform in Java
- [ ] Support pipeline option to auto-replace GBEK with encrypted GBEK
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [x] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3321040104
> 2. Enforcing that you only persist GBK'd data (this is how most runners work/checkpoint already AFAIK).
The Schrodinger SeamRunner doesn't work this way. Many of the SeamRunner stage boundaries are just from GBKs but there are many stage boundaries that arise from incompatible environments as well. Not sure how other runners handle stage boundaries - do they just pipe together data streams directly between workers?
> An alternative would be to eventually offer both and let the runner choose the mode that works for them.
If we had an API that specifies how secrets are determined as a PipelineOption, then only the only-GBK-persisting runners could replace GBKs with GBEKs and other runners could use the secret in all places where they persist pcollections.
I think a main concern here for me is the addition of a new transform that blurs the SDK/runner boundary. When onboarding new developers onto Beam, the biggest hurdle I face is introducing `Reshuffle` since it breaks the promise of "As a pipeline author, you don't have to worry about the _how_ of execution" and GBEK may be another transform like `Reshuffle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3320859785
Would it be feasible to implement this capability generically for all transforms using a wrapper secret coder? It'd be a type of `BytesCoder` that allows specifying a secret. Then runners could use this secret coder and wrap any final bytes coder they'd be using to read/write for the SKD harness and GBK without requiring users to adjust their pipelines to accommodate encryption-at-rest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3320793406
Good question - I updated the description a bit. Basically, the goal here is to allow runners to add this capability; in order to do so, they'd need to guarantee that they're not going to materialize any PCollections in an undesirable way outside of the context of a GBK.
But using the transform with an arbitrary runner isn't necessarily enough by itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3320722989
Just passing by and saw this ticket. Out of curiosity, how are runners supposed to guarantee encryption at rest? Can't a runner arbitrarily decide when to materialize pcollections to disk? In which case, would the runner need to be able to encrypt at any point of the pipeline?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3321292562
> Not sure how other runners handle stage boundaries - do they just pipe together data streams directly between workers?
Generally this reduces to a reshuffle or similar which eventually uses GBK under the hood. But definitely not part of the Beam API.
> If we had an API that specifies how secrets are determined as a PipelineOption, then only the only-GBK-persisting runners could replace GBKs with GBEKs and other runners could use the secret in all places where they persist pcollections.
Yeah, this is more or less what I'm suggesting here; this issue is more of an implementation detail (though potentially you could only encrypt some of your pipeline at the GBK step if you want to use it directly I guess).
Note that even with this transform and the GBK guarantees, it is still impossible for users to get encryption at rest guarantees since they would need to replace GBK in reshuffles (and other transforms) somehow, and they won't be able to do that. So a pipeline option will be necessary regardless.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3405777931
At this point, all outstanding items are complete, so closing here. We can follow up on https://github.com/apache/beam/issues/36214#issuecomment-3321040104 separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3320939961
That would definitely be an option. I'll note that it is probably less efficient though, since you need to apply it to any encoding/decoding, as opposed to just:
1) Applying it at the GBK boundary
2) Enforcing that you only persist GBK'd data (this is how most runners work/checkpoint already AFAIK).
An alternative would be to eventually offer both and let the runner choose the mode that works for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "hjtran (via GitHub)" <gi...@apache.org>.
hjtran commented on issue #36214:
URL: https://github.com/apache/beam/issues/36214#issuecomment-3321316001
Ah gotcha. This all makes sense to me then. Thanks for taking the time to explain!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: Add GroupByEncryptedKey transform [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #36214: [Feature Request]: Add GroupByEncryptedKey transform
URL: https://github.com/apache/beam/issues/36214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org