[jira] [Commented] (TEZ-793) Staging directories accumulate and pollute the home directory

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891855#comment-13891855 ] 

Bikas Saha commented on TEZ-793:
--------------------------------

Tez does not come with a default staging dir. The user is expected to specify it via config. However, whatever Tez puts in that staging dir (for now its the dag plan file) should be deleted by Tez.
I believe we had a jira to make Tez clean up the files it uploads to the staging directory. I cannot find it. Perhaps we can rename this jira to for that purpose.

> Staging directories accumulate and pollute the home directory
> -------------------------------------------------------------
>
>                 Key: TEZ-793
>                 URL: https://issues.apache.org/jira/browse/TEZ-793
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Mark Wagner
>
> Right now, most of the examples use 
> {noformat}
> Path stagingDir = new Path(fs.getWorkingDirectory(), UUID.randomUUID().toString());
> {noformat}
> to get the staging directory. This results in home directories filled with directories that have to be manually deleted one by one. The MRRSleep example is the only example which uses TEZ_AM_STAGING_DIR, but that defaults to '/tmp/tez/staging', which breaks for more than one user.
> It would be good to have a standardized way of handling staging directories for Tez. Optimally, it would be nice if Tez would handle the lifecycle of its staging directory itself.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)