[jira] [Commented] (TEZ-2872) Tez AM can be overwhelmed by TezTaskUmbilicalProtocol.getTask responses

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949368#comment-14949368 ] 

Jason Lowe commented on TEZ-2872:
---------------------------------

This is similar to a scenario MapReduce encountered before, see MAPREDUCE-4730.  One could argue the problem can be solved by fixing HADOOP-8942 since the lack of flow control in the RPC layer is what allows the responses to accumulate without limit and hit the OOM.  While that would prevent the OOM condition, it would leave the AM in a state where its outbound network is saturated, unable to respond properly to other RPC requests.  Not exactly an ideal scenario from the perspective of others trying to talk to the AM.

We may want to consider adding a rate throttle to getTask processing for situations like this.  It would be nice if we could cheaply calculate the size of the ContainerTask response and only send out some configurable MB/sec of responses, but ContainerTask is a writable, not a protobuf, so we don't know the size of it until we actually serialize it to a buffer in the depths of the RPC layer.  Maybe we could scrape the ContainerTask for all the additional resources and user payloads squirreled away everywhere and do a quick guesstimate on the total size as a proxy.  Or we could just do a simple RPC count and only allow so many per second.

> Tez AM can be overwhelmed by TezTaskUmbilicalProtocol.getTask responses
> -----------------------------------------------------------------------
>
>                 Key: TEZ-2872
>                 URL: https://issues.apache.org/jira/browse/TEZ-2872
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jason Lowe
>
> When a large job runs on a large cluster with a large user payload then the AM can end up hitting OOM conditions.  For example, Pig-on-Tez can require a significant user payload (approaching 1MB) for vertices, inputs, and outputs in the DAG.  This can cause the ContainerTask response to be rather large per task, which can lead to a situation where the AM is generating output faster than the network interface can process it.  If there are enough containers asking for tasks then this leads to an OOM condition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)