You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2014/08/01 20:23:38 UTC
[jira] [Commented] (TEZ-1337) Handling of local-dirs for Local Mode
[ https://issues.apache.org/jira/browse/TEZ-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14082679#comment-14082679 ]
Siddharth Seth commented on TEZ-1337:
-------------------------------------
Marking this as a blocker.
Using the staging directory has several issues
1) The staging dir may not be known up front, which means data cannot exist in this dir. Instead users would have to provide an absolutely path instead of a relative path. (e.g. /user/${user}/input generally just needs to be specified as 'input' for most examples).
2) The staging dir (dag-specific) is/will-be cleaned up by Tez itself if it creates is. Many of the examples also clean up the staging directory.
[~airbots] I think we should make the following changes.
- Introduce a local dir for local mode. This will be the base directory for the filesystem that is being used for local mode.
- The staging directory, however, must continue to work since the Client writes into this and the AM reads contents back.
- Fallback to using the base staging-dir if this property is not configured. The root path would end up being the root-staging dir, and not the application specific staging directory, since that is likely to be deleted.
> Handling of local-dirs for Local Mode
> -------------------------------------
>
> Key: TEZ-1337
> URL: https://issues.apache.org/jira/browse/TEZ-1337
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Chen He
>
> staging dir is being used to write intermediate data. At some point, it may be worthwhile to confgure a separate work area. IAC, these should be cleaned up - at least the intermediate data after a local session executes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)