You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Saikat (JIRA)" <ji...@apache.org> on 2015/09/22 23:36:04 UTC
[jira] [Comment Edited] (TEZ-2850) Tez MergeManager OOM for small
Map Outputs
[ https://issues.apache.org/jira/browse/TEZ-2850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903481#comment-14903481 ]
Saikat edited comment on TEZ-2850 at 9/22/15 9:35 PM:
------------------------------------------------------
This is a unique scenario, that we faced, while running a Tez Job.
A reducer vertex task fetches around 200000 map outputs, each of around ~100 odd bytes.
So total mapoutput size is around 200000 * 100 ~ 20Mb.
The MergeManager has a merge threshold check, where if it crosses this threshold, InmemoryMerger will be triggered and it will merge & spill the inmemory fetched map outputs to disk to free up memory.
In our scenario, mergethreshold(~500mb) >> commitMemory(~20mb), So inMemory merger never gets triggerd.
Finally when the finalMerge() is called in close(), MergeManager calls createInMemorySegments() to do the final merge.
In this, when Tez creates a IFileInputStream object for the InMemoryReader, the IFileInputStream allocates a buffer of size 4096(hard coded).
Thus the total size of a single inmemory segment comes to around 5kb, even though data in this segment is only in order of 100 bytes. So, for 200000 map outputs, the total size is 200000 * 5000 ~ 1G, which causes OOM!
Attached is a snapshot of the heap dump which shows this scenario.
was (Author: saikatr):
This is a unique scenario, that we faced, while running a Tez Job.
A reducer vertex task fetches around 200000 map outputs, each of around ~100 odd bytes.
So total mapoutput size is around 200000 * 100 ~ 20Mb.
The MergeManager has a merge threshold check, where if it crosses this threshold, InmemoryMerger will be triggered and it will spill the inmemory fetched map outputs to disk to free up memory.
In our scenario, mergethreshold(~500mb) >> commitMemory(~20mb), So inMemory merger never gets triggerd.
Finally when the finalMerge() is called in close(), MergeManager calls createInMemorySegments() to do the final merge.
In this, when Tez creates a IFileInputStream object for the InMemoryReader, the IFileInputStream allocates a buffer of size 4096(hard coded).
Thus the total size of a single inmemory segment comes to around 5kb, even though data in this segment is only in order of 100 bytes. So, for 200000 map outputs, the total size is 200000 * 5000 ~ 1G, which causes OOM!
Attached is a snapshot of the heap dump which shows this scenario.
> Tez MergeManager OOM for small Map Outputs
> ------------------------------------------
>
> Key: TEZ-2850
> URL: https://issues.apache.org/jira/browse/TEZ-2850
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Saikat
> Attachments: OOM_1.png, OOM_2.png, OOM_3.png
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)