You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Jingsong Lee (Jira)" <ji...@apache.org> on 2019/12/10 09:25:00 UTC

[jira] [Created] (FLINK-15172) optimize the operator algorithm to lazily allocate memory

Jingsong Lee created FLINK-15172:
------------------------------------

             Summary: optimize the operator algorithm to lazily allocate memory
                 Key: FLINK-15172
                 URL: https://issues.apache.org/jira/browse/FLINK-15172
             Project: Flink
          Issue Type: Sub-task
          Components: Table SQL / Runtime
            Reporter: Jingsong Lee
             Fix For: 1.11.0


Now after FLINK-14063 , operators will get all manage memory of TaskManager,  The cost of over allocate memory is very high, lead to performance regression of small batch sql jobs:
 * Allocate memory will have the cost of memory management algorithm.
 * Allocate memory will have the cost of memory initialization, will set all memory to zero. And this initialization will require the operating system to actually allocate physical memory.

 * Over allocate memory will squash the file cache too.

We can optimize the operator algorithm, apply lazy allocation, and avoid meaningless memory allocation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)