[jira] [Updated] (TEZ-3363) Delete intermediate data at the vertex level for Shuffle Handler

["Kuhu Shukla (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kuhu Shukla updated TEZ-3363:
-----------------------------
    Attachment: TEZ-3363.001.patch

First cut of the patch.

This patch adds a new event that is sent by every vertex to its ancestors at a given/configurable height. The ancestors are precomputed during DagPlan creation. When a vertex successfully completes, it sends this event to all the ancestors. A vertex that receives this event, calls vertexComplete() from the DagAppMaster all the way down to the ContainerLauncher and boiling down to the DeletionTracker. The vertex delete http calls to shuffle handler are sent serially right now to the nodes/shufflehandlers and that can be changed based on some feedback by sharing the threadpool or creating a new one. The shuffle handler does a regex match on the directory names and deletes the ones belonging to the passed vertex.
In case a vertex fails vertex deletion does not quite kick in since the DAG will be failed and cleanup will follow shorty anyway.
ShuffleUtils.isTezShuffleHandler calls have been added to VertexImpl which therefore has to import this new class, which may be undesired( ? ).
Will let the precommit run and ask for some feedback soon.

> Delete intermediate data at the vertex level for Shuffle Handler
> ----------------------------------------------------------------
>
>                 Key: TEZ-3363
>                 URL: https://issues.apache.org/jira/browse/TEZ-3363
>             Project: Apache Tez
>          Issue Type: Sub-task
>            Reporter: Jonathan Eagles
>            Assignee: Kuhu Shukla
>         Attachments: TEZ-3363.001.patch
>
>
> For applications like pig where processing times can be very long, applications may choose to delete intermediate data for a sub dag. For example if a DAG has synced data to HDFS, all upstream intermediate data can be safely deleted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)