You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/07/09 20:35:53 UTC

[GitHub] [beam] boyuanzz commented on pull request #14811: [BEAM-11996] spannerio splittable

boyuanzz commented on pull request #14811:
URL: https://github.com/apache/beam/pull/14811#issuecomment-877442652


   Double checked the `SpannerIO` implementation. It seems like the current implementation is **not**  built on top of `BoundedSource`. The read is expanded into: Create() -> GeneratePartitions -> Reshuffle -> DoFn reading from a partition. And it seems like we are not able to split inside one partition(I'm not 100% sure whether it's true).
   
    So the value of converting this read into `SplittableDoFn` is:
   * Taking advantage of expansion of SplittableDoFn. That means we can save the expansion of read to `Create() -> SplittableDoFn()`. But it's not fully true before resolving https://issues.apache.org/jira/browse/BEAM-12522.
   
   @chamikaramj Do you think it's still valuable to convert SpannerIO into SplittalbeDoFn? Or do I miss anything there?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org