[jira] [Commented] (TEZ-3954) Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords

["TezQA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16514699#comment-16514699 ] 

TezQA commented on TEZ-3954:
----------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment
  http://issues.apache.org/jira/secure/attachment/12928069/TEZ-3954.001.patch
  against master revision 3f2373e.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version 3.0.1) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in :
                   org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2838//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2838//console

This message is automatically generated.


> Reduce Tez Shuffle Handler Memory needs for holding TezIndexRecords
> -------------------------------------------------------------------
>
>                 Key: TEZ-3954
>                 URL: https://issues.apache.org/jira/browse/TEZ-3954
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Major
>         Attachments: TEZ-3954.001.patch
>
>
> Tez Shuffle Handler holds on to entire SpillRecord to better accommodate for range fetch operation. For single reduce fetch holding on to spillRecord can overwhelm memory if Index Cache is overrun. In the case of single fetch, only keep the single tez index record needed and for range fetch, keep the entire spill record in memory, but free it after it is needed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)