You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/03/06 11:16:40 UTC
[jira] [Commented] (FLINK-3322) MemoryManager creates too much GC
pressure with iterative jobs
[ https://issues.apache.org/jira/browse/FLINK-3322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15182076#comment-15182076 ]
ASF GitHub Bot commented on FLINK-3322:
---------------------------------------
GitHub user Xazax-hun opened a pull request:
https://github.com/apache/flink/pull/1769
[FLINK-3322] MemoryManager creates too much GC pressure with iterative jobs.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Xazax-hun/flink MemoryManager
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1769.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1769
----
commit 84e43270415b1a83590e374b6a5440e73e91dad1
Author: Gabor Horvath <xa...@gmail.com>
Date: 2016-03-05T08:37:32Z
[FLINK-3322] Added the test case of ggevay to reproduce the performance
issue.
commit 3fcce08cffae95f48855b58a576603712f522e67
Author: Gabor Horvath <xa...@gmail.com>
Date: 2016-03-05T13:36:31Z
[FLINK-3322] MemoryManager creates too much GC pressure with iterative jobs.
----
> MemoryManager creates too much GC pressure with iterative jobs
> --------------------------------------------------------------
>
> Key: FLINK-3322
> URL: https://issues.apache.org/jira/browse/FLINK-3322
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
> Affects Versions: 1.0.0
> Reporter: Gabor Gevay
> Assignee: Gabor Horvath
> Priority: Critical
> Fix For: 1.0.0
>
>
> When taskmanager.memory.preallocate is false (the default), released memory segments are not added to a pool, but the GC is expected to take care of them. This puts too much pressure on the GC with iterative jobs, where the operators reallocate all memory at every superstep.
> See the following discussion on the mailing list:
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Memory-manager-behavior-in-iterative-jobs-tt10066.html
> Reproducing the issue:
> https://github.com/ggevay/flink/tree/MemoryManager-crazy-gc
> The class to start is malom.Solver. If you increase the memory given to the JVM from 1 to 50 GB, performance gradually degrades by more than 10 times. (It will generate some lookuptables to /tmp on first run for a few minutes.) (I think the slowdown might also depend somewhat on taskmanager.memory.fraction, because more unused non-managed memory results in rarer GCs.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)