[GitHub] [beam] lukecwik commented on pull request #17783: [BEAM-14534] Allow users to compress values being shuffled in dataflow

[GitBox <gi...@apache.org>]

lukecwik commented on PR #17783:
URL: https://github.com/apache/beam/pull/17783#issuecomment-1143765738

   I see that BEAM-14534 does specifically mention using a wrapping coder to compress the data and I think it would still be a better solution since it would allow you to:
   * plugin compression into more places such as compressing the shuffle key, user state and side inputs
   * works for all runners and not just Dataflow runner v1. (Note that this solution won't work for any portable runners like Dataflow runner v2 since you can't edit the shuffle code)
   * the compression can be configured per GroupByKey since in some places it won't make sense since the values are not easily compressible
   
   You can also create a PipelineVisitor that walks all transforms and replaces the `KV<KeyCoder, ValueCoder>` with `KV<KeyCoder, CompressingCoder<ValueCoder>>` so that users don't need to do this themselves which would give similar behavior as specifying the shuffle compressor.
   
   The other part of the change makes sense to reduce byte[] copies by using ByteString.
   
   CC: @tudorm


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org