You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Maximilian Michels (Jira)" <ji...@apache.org> on 2020/09/08 14:36:00 UTC

[jira] [Commented] (BEAM-10760) Cleanup timers lead to unbounded state accumulation in global window

    [ https://issues.apache.org/jira/browse/BEAM-10760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17192247#comment-17192247 ] 

Maximilian Michels commented on BEAM-10760:
-------------------------------------------

Fixed via 5fd0010e52c5efee36760c302b03181e429da02f and e010b15ba16e7867cb1acd36d9215674606a7d2a.

> Cleanup timers lead to unbounded state accumulation in global window
> --------------------------------------------------------------------
>
>                 Key: BEAM-10760
>                 URL: https://issues.apache.org/jira/browse/BEAM-10760
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core, runner-flink
>    Affects Versions: 2.21.0
>            Reporter: Thomas Weise
>            Assignee: Thomas Weise
>            Priority: P2
>             Fix For: 2.25.0
>
>          Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> For each key, the runner sets a cleanup timer that is designed to garbage collect state at the end of a window. For a global window, these timers will stay around until the pipeline terminates. Depending on the key cardinality, this can lead to unbounded state growth, which in the case of the Flink runner is observable in the growth of checkpoint size.
> https://lists.apache.org/thread.html/rae268806035688b77646195505e5b7a56568a38feb1e52d6341feedd%40%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)