[jira] [Commented] (TEZ-3877) Delete unordered spill files once merge is done

["TezQA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314116#comment-16314116 ] 

TezQA commented on TEZ-3877:
----------------------------

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment
  http://issues.apache.org/jira/secure/attachment/12904866/TEZ-3877.001.patch
  against master revision d777f45.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version 3.0.1) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/2708//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/2708//console

This message is automatically generated.

> Delete unordered spill files once merge is done
> -----------------------------------------------
>
>                 Key: TEZ-3877
>                 URL: https://issues.apache.org/jira/browse/TEZ-3877
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Jason Lowe
>         Attachments: TEZ-3877.001.patch
>
>
>   I see that spill files are not deleted right after merge completes. We should do that as it takes up a lot of space and we can't afford that wastage when Tez takes up a lot of shuffle space with complex DAGs. [~jlowe] told me they are only cleaned up after application completes as they are written in app directory and not container directory. That also has to be done so that they are cleaned up by node manager during task failures or container crashes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)