[jira] [Created] (PIG-4626) [Pig on Tez] OOM in case of multiple outputs and POPartialAgg

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

Rohini Palaniswamy created PIG-4626:
---------------------------------------

             Summary: [Pig on Tez] OOM in case of multiple outputs and POPartialAgg
                 Key: PIG-4626
                 URL: https://issues.apache.org/jira/browse/PIG-4626
             Project: Pig
          Issue Type: Bug
            Reporter: Rohini Palaniswamy
             Fix For: 0.16.0, 0.15.1


  tez.task.scale.memory.reserve-fraction is 0.3 by default which assumes 70% of memory is available for io.sort.mb. If map is configured as 1G Xmx and io.sort.mb as 256MB, and there is group by on 3 different keys it has 3 different outputs and Tez's WeightedScalingMemoryDistributor allocates 256MB to each of the outputs which leaves very less memory for Pig and POPartialAgg (which requires 20% of memory) and sometime leads to OOM. This also causes SpillableMemoryManager to be invoked often as its threshold is set to 70% of memory.

Need to set tez.task.scale.memory.reserve-fraction as 0.5 for all pig jobs and 0.4 in case there is POPartialAgg in combiner plan.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)