[GitHub] [beam] kennknowles opened a new issue, #18459: Improve scalability of the Watch transform

[GitBox <gi...@apache.org>]

kennknowles opened a new issue, #18459:
URL: https://github.com/apache/beam/issues/18459

   [https://github.com/apache/beam/pull/3565](https://github.com/apache/beam/pull/3565) introduces the Watch transform [http://s.apache.org/beam-watch-transform](http://s.apache.org/beam-watch-transform).
   
   The implementation leaves several scalability-related TODOs:
    1) The state stores hashes and timestamps of outputs that have already been output and should be omitted from future polls. We could garbage-collect this state, e.g. dropping elements from "completed" and from addNewAsPending() if their timestamp is more than X behind the watermark.
    2) When a poll returns a huge number of elements, we don't necessarily have to add all of them into state.pending - instead we could add only N oldest elements and ignore others, relying on future poll rounds to provide them, in order to avoid blowing up the state. Combined with garbage collection of GrowthState.completed, this would make the transform scalable to very large poll results.
   
   Imported from Jira [BEAM-2680](https://issues.apache.org/jira/browse/BEAM-2680). Original Jira may contain additional context.
   Reported by: jkff.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] zhengbuqian commented on issue #18459: Improve scalability of the Watch transform

["zhengbuqian (via GitHub)" <gi...@apache.org>]

zhengbuqian commented on issue #18459:
URL: https://github.com/apache/beam/issues/18459#issuecomment-1485848276

   This has caused a dataflow customer to experience error: too many hashes/timestamps stored causing [this ValueState](https://github.com/apache/beam/blob/07f52a478174f8733c7efedb7189955142faa5fa/runners/core-java/src/main/java/org/apache/beam/runners/core/SplittableParDoViaKeyedWorkItems.java#L258) to exceed max allowed single value size limit of 80MB(https://cloud.google.com/dataflow/quotas#limits).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org