You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Akshay Iyangar (Jira)" <ji...@apache.org> on 2019/09/11 18:38:00 UTC

[jira] [Created] (BEAM-8212) StatefulParDoFn creates GC timers for every record

Akshay Iyangar created BEAM-8212:
------------------------------------

             Summary: StatefulParDoFn creates GC timers for every record 
                 Key: BEAM-8212
                 URL: https://issues.apache.org/jira/browse/BEAM-8212
             Project: Beam
          Issue Type: Bug
          Components: beam-community
            Reporter: Akshay Iyangar
            Assignee: Aizhamal Nurmamat kyzy


Hi 

So currently the StatefulParDoFn create timers for all the records.
[https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/StatefulDoFnRunner.java#L211]

This becomes a problem if you are using GlobalWindows for streaming where these timers get created and never get closed since the window will never close.

This is a problem especially if your memory bound in rocksDB where these timers take up potential space and sloe the pipelines considerably.

Was wondering that if the pipeline runs in global windows we should avoid adding timers to it at all?

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)