[jira] [Created] (BEAM-12335) Apply basic fusion to Java DirectRunner to avoid keeping all intermittent results in memory

["Boyuan Zhang (Jira)" <ji...@apache.org>]

Boyuan Zhang created BEAM-12335:
-----------------------------------

             Summary: Apply basic fusion to Java DirectRunner to avoid keeping all intermittent results in memory 
                 Key: BEAM-12335
                 URL: https://issues.apache.org/jira/browse/BEAM-12335
             Project: Beam
          Issue Type: Improvement
          Components: runner-direct
            Reporter: Boyuan Zhang


Current java direct runner doesn't fuse transforms into steps. Instead, it almost executes each transform one by one. It results in memory pressure when any transform is high-fanout.

We already have a simple fusion logic in Java SDK(https://github.com/apache/beam/blob/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPipelineFuser.java). Work remaining here might be:
* Apply such fusion into DirectRunner
* Change the DirectRunner to be able run the fused steps.

I understand that DirectRunner doesn't expect processing large volume data and changing DirectRunner execution might be a fair amount of work.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)