[jira] [Created] (BEAM-8848) Flink: Memory efficient GBK implementation for batch runner

["David Moravek (Jira)" <ji...@apache.org>]

David Moravek created BEAM-8848:
-----------------------------------

             Summary: Flink: Memory efficient GBK implementation for batch runner
                 Key: BEAM-8848
                 URL: https://issues.apache.org/jira/browse/BEAM-8848
             Project: Beam
          Issue Type: Improvement
          Components: runner-flink
            Reporter: David Moravek
            Assignee: David Moravek


In current batch runner, all the values for a single key need to fit in memory, because the resulting GBK iterable is materialized using "List" data structure.

Implications:
- This blocks user from using custom sharding in most of the IOs, as the whole shard needs to fit in memory.
- Frequent OOM failures in case of skewed data (pipeline should be running slow instead of failing). This is super hard to debug for inexperienced user.

We can do way better for non-merging windows, the same way we do for Spark runner. Only drawback is, that this implementation does not support result re-iterations. We'll support turning this implementation on and off, if user needs to trade off reiterations for memory efficiency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)