[jira] [Commented] (BEAM-2680) Improve scalability of the Watch transform

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17179860#comment-17179860 ] 

Beam JIRA Bot commented on BEAM-2680:
-------------------------------------

This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3.

Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean.


> Improve scalability of the Watch transform
> ------------------------------------------
>
>                 Key: BEAM-2680
>                 URL: https://issues.apache.org/jira/browse/BEAM-2680
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Eugene Kirpichov
>            Priority: P2
>              Labels: stale-P2
>
> [https://github.com/apache/beam/pull/3565] introduces the Watch transform [http://s.apache.org/beam-watch-transform].
> The implementation leaves several scalability-related TODOs:
>  1) The state stores hashes and timestamps of outputs that have already been output and should be omitted from future polls. We could garbage-collect this state, e.g. dropping elements from "completed" and from addNewAsPending() if their timestamp is more than X behind the watermark.
>  2) When a poll returns a huge number of elements, we don't necessarily have to add all of them into state.pending - instead we could add only N oldest elements and ignore others, relying on future poll rounds to provide them, in order to avoid blowing up the state. Combined with garbage collection of GrowthState.completed, this would make the transform scalable to very large poll results.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)