You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Ivan Mitic (JIRA)" <ji...@apache.org> on 2012/08/27 20:37:08 UTC
[jira] [Created] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Ivan Mitic created HADOOP-8734:
----------------------------------
Summary: LocalJobRunner does not support private distributed cache
Key: HADOOP-8734
URL: https://issues.apache.org/jira/browse/HADOOP-8734
Project: Hadoop Common
Issue Type: Bug
Components: filecache
Reporter: Ivan Mitic
Assignee: Ivan Mitic
It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454539#comment-13454539 ]
Bikas Saha commented on HADOOP-8734:
------------------------------------
So if I understand this right, this fixes a generic deficiency in LocalJobRunner which wasnt showing up because by default files are public to read on Linux FS and so LocalJobRunner would not see issues in accessing private distributed cache from the local FS.
Also, this would make the change to TestMRWithDistributedCache unnecessary?
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli resolved HADOOP-8734.
---------------------------------------------
Resolution: Fixed
Fix Version/s: 1-win
Hadoop Flags: Reviewed
+1, looks good. Verified that the test fails without the code change and passes with.
Just committed this to branch-1-win. Thanks Ivan!
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Fix For: 1-win
>
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454943#comment-13454943 ]
Ivan Mitic commented on HADOOP-8734:
------------------------------------
Thanks Bikas.
bq. So if I understand this right, this fixes a generic deficiency in LocalJobRunner which wasnt showing up because by default files are public to read on Linux FS and so LocalJobRunner would not see issues in accessing private distributed cache from the local FS.
Correct, this is how I see the problem.
bq. Also, this would make the change to TestMRWithDistributedCache unnecessary?
Given that I'm making a bug fix I should also add a test case that catches the bug. In this case, it was enough to slightly modify one test to catch the bug. Make sense?
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Mitic updated HADOOP-8734:
-------------------------------
Attachment: HADOOP-8734-LocalJobRunner.patch
Attaching the fix proposal.
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455043#comment-13455043 ]
Bikas Saha commented on HADOOP-8734:
------------------------------------
Sorry. I got totally confused and misread the test file name in the patch. +1. Thanks!
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454326#comment-13454326 ]
Bikas Saha commented on HADOOP-8734:
------------------------------------
bq. Check out the fix I did to TestMRWithDistributedCache, this is an E2E use case.
What fix are you mentioning?
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449703#comment-13449703 ]
Ivan Mitic commented on HADOOP-8734:
------------------------------------
Thanks for reviewing Bikas.
bq.Can you please elaborate on the cause and the fix?
The problem I see is that LocalJobRunner does not work with private distributed cache files. Private DC files are files that are not visible to "others" on the LFS leading to them being marked as private in DC. Check out the fix I did to TestMRWithDistributedCache, this is an E2E use case. Specifically, {{testLocalJobRunner}} now fails without my code fix.
The fix is in the LocalJobRunner to explicitly localize the private cache (I saw this pattern being used for private DC cache localization in some tests).
This is something that is nice to have especially on Windows, as all files are private by default (not visible to "others"). HADOOP-8731 has some info on DC on Windows that might be helpful.
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13456048#comment-13456048 ]
Ivan Mitic commented on HADOOP-8734:
------------------------------------
Awesome, thanks!
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Fix For: 1-win
>
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Ivan Mitic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454336#comment-13454336 ]
Ivan Mitic commented on HADOOP-8734:
------------------------------------
What I mean is that I made a small change to TestMRWithDistributedCache such that the test fails without my fix to the LocalJobRunner.
{code}
+ // Change permissions on one file to be private (others cannot read
+ // the file) to make sure private distributed cache works fine with
+ // the LocalJobRunner.
+ FileUtil.chmod(fourth.toUri().getPath(), "700");
{code}
Let me know if this clarifies things.
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8734) LocalJobRunner does not support
private distributed cache
Posted by "Bikas Saha (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444554#comment-13444554 ]
Bikas Saha commented on HADOOP-8734:
------------------------------------
Can you please elaborate on the cause and the fix? Thanks!
> LocalJobRunner does not support private distributed cache
> ---------------------------------------------------------
>
> Key: HADOOP-8734
> URL: https://issues.apache.org/jira/browse/HADOOP-8734
> Project: Hadoop Common
> Issue Type: Bug
> Components: filecache
> Reporter: Ivan Mitic
> Assignee: Ivan Mitic
> Attachments: HADOOP-8734-LocalJobRunner.patch
>
>
> It seems that LocalJobRunner does not support private distributed cache. The issue is more visible on Windows as all DC files are private by default (see HADOOP-8731).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira