[GitHub] [beam] reuvenlax commented on pull request #15256: Revert "[BEAM-11934] Remove Dataflow override of streaming WriteFiles with runner determined sharding"

[GitBox <gi...@apache.org>]

reuvenlax commented on pull request #15256:
URL: https://github.com/apache/beam/pull/15256#issuecomment-891198489


   I'm not convinced that the flakiness was caused by this PR, as it happens
   even when this PR is rolled back.
   
   I'm still confused about how WordCount ever succeeds after this PR was
   submitted as it writes to files, it does not specify numShards, and it also
   runs on windmill appliance.
   
   On Mon, Aug 2, 2021 at 10:18 AM Siyuan Chen ***@***.***>
   wrote:
   
   > If I understand it correctly the flakiness was caused by running a
   > streaming pipeline on a bounded source in the word count pipeline. Note
   > that the implementation for runner determined sharding in the WriteFiles
   > is different for bounded and unbounded source. So the override was only
   > used by bounded source in streaming and without the override the word count
   > pipeline unexpectedly picked up the implementation for batch.
   >
   > Perhaps a better fix might be to modify the override to also check the
   > bounded-ness of the input so unbounded datacan still adopt proper runner
   > determined sharding implementation for streaming.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/beam/pull/15256#issuecomment-891195968>, or
   > unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AFAYJVKIN6KIRFV66I2WC63T23HOTANCNFSM5BJU4KJQ>
   > .
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org