You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramya Sunil (JIRA)" <ji...@apache.org> on 2011/07/29 22:46:09 UTC
[jira] [Created] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
[MR-279] Redundant "file:" directory created in appcache
---------------------------------------------------------
Key: MAPREDUCE-2757
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Priority: Minor
Fix For: 0.23.0
A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090751#comment-13090751 ]
Siddharth Seth commented on MAPREDUCE-2757:
-------------------------------------------
Jonathan, the path is still being created when running unit tests - TestResourceLocalization seems to be the culprit.
It may make sense to generate a warning, if not an error, until LocalDirAllocator works with schemes.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated MAPREDUCE-2757:
---------------------------------------
Status: Patch Available (was: Open)
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102423#comment-13102423 ]
Arun C Murthy commented on MAPREDUCE-2757:
------------------------------------------
Jonathan, is this ready for commit? I don't see it marked PA, thanks.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2757:
-------------------------------------
Status: Open (was: Patch Available)
Again the Apache headers :(.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092030#comment-13092030 ]
Hadoop QA commented on MAPREDUCE-2757:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12491544/MAPREDUCE-2757-trunk-v1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/543//console
This message is automatically generated.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073019#comment-13073019 ]
Jonathan Eagles commented on MAPREDUCE-2757:
--------------------------------------------
It looks like the "file:" directory is generated in ContainerLocalizer.initDirs.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Priority: Minor
> Fix For: 0.23.0
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2757:
-------------------------------------
Priority: Blocker (was: Minor)
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13091298#comment-13091298 ]
Siddharth Seth commented on MAPREDUCE-2757:
-------------------------------------------
For now, we should probably just have a utility function to figure out how to interpret paths specified by the user - this can take care of converting them to the correct format or spewing out warnings (ref: DataNode.getStorageDirs). Having a consistent format for specifying local dirs between MR and HDFS would be nice.
The LocalDirAllocator could be changed to work with and without schemes (accepting only file:) while still working with absolute and relative paths. That would be a hadoop common jira.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090460#comment-13090460 ]
Jonathan Eagles commented on MAPREDUCE-2757:
--------------------------------------------
Further analysis
ContainerLocalizer.initDirs qualifies (e.g. file:///tmp instead of just /tmp) all paths and then serializes the qualified path as strings into the Configuration. Qualified paths are not part of the supported interface as part of the fs LocalDirAllocator. It in turn tries to deserializes the qualified path strings directly into a java.io.File. Thus instead of testing to see whether the path is user writable, it instead recursively creates a dir name file: containing a dir named tmp in the current working directory of the local file system.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles reassigned MAPREDUCE-2757:
------------------------------------------
Assignee: Jonathan Eagles
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli resolved MAPREDUCE-2757.
------------------------------------------------
Resolution: Duplicate
bq. [..] should be duplicated to HADOOP-7575, which fixes this problem more systematically.
+1. Closing this.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13102915#comment-13102915 ]
Jonathan Eagles commented on MAPREDUCE-2757:
--------------------------------------------
@Arun
My feeling is that JIRA should be duplicated to HADOOP-7575, which fixes this problem more systematically. If you agree, I will go ahead and dupe this JIRA.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles updated MAPREDUCE-2757:
---------------------------------------
Attachment: MAPREDUCE-2757-trunk-v1.patch
No unit test were added for this one line change.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2757) [MR-279] Redundant "file:"
directory created in appcache
Posted by "Jonathan Eagles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090757#comment-13090757 ]
Jonathan Eagles commented on MAPREDUCE-2757:
--------------------------------------------
Good catch! It looks like it is also using qualified paths in the tests. I started a patch for the local dir allocator allowing qualified paths. My hesitation is that after starting the patch for allowing schemas, local dir allocator stopped allowing relative paths. Do you have any suggestion on what direction to go. It seems that both can help the problem. Please advise.
> [MR-279] Redundant "file:" directory created in appcache
> ---------------------------------------------------------
>
> Key: MAPREDUCE-2757
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2757
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2757-trunk-v1.patch
>
>
> A redundant directory called "file:" is being created under ${yarn.server.nodemanager.local-dir}/usercache/${username}/appcache/appID which further has a directory structure ${yarn.server.nodemanager.local-dir}/usercache/${username}/(appcache|filecache)/appID/filecache which is empty.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira