You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/08/02 08:52:26 UTC

[GitHub] [beam] nielm commented on pull request #14811: [BEAM-11996] spannerio splittable

nielm commented on pull request #14811:
URL: https://github.com/apache/beam/pull/14811#issuecomment-890849612


   > Are you sure that this splitting implementation is beneficial ?
   > 
   > I think what it actually does is disabling (or reducing) partition-based initial splitting and pushing partition based splitting to dynamic splitting.
   
   I agree with Cham here, 
   From a high-level view, this PR replaces: 
   
   - a 1->many fan-out - GeneratePartitionsFn  \
     (where in practice "many" is only a few 10s of elements, not millions)
   - followed by a Reshuffle
   
   with a single splittable DoFn
   
   In my opinion this will not increase parallelization, and makes the code more confusing/verbose.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org