You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Patrick Hunt (JIRA)" <ji...@apache.org> on 2012/11/21 18:59:59 UTC
[jira] [Created] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Patrick Hunt created MRUNIT-161:
-----------------------------------
Summary: some distributed cache apis not working - JobContext getCacheArchives()
Key: MRUNIT-161
URL: https://issues.apache.org/jira/browse/MRUNIT-161
Project: MRUnit
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Patrick Hunt
Priority: Blocker
Fix For: 1.0.0
I'm using the new MRUnit method withCacheArchive available in 1.0.0.
DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MRUNIT-161) some distributed cache apis
not working - JobContext getCacheArchives()
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503508#comment-13503508 ]
Brock Noland edited comment on MRUNIT-161 at 11/25/12 4:27 PM:
---------------------------------------------------------------
[~dbeech] Do you know the difference is between the getLocal*Cache* and getCache* methods? It looks like they are using a different configuration key.
was (Author: brocknoland):
@dbeech Do you know the difference is between the getLocal*Cache* and getCache* methods? It looks like they are using a different configuration key.
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MRUNIT-161) some distributed cache apis
not working - JobContext getCacheArchives()
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503508#comment-13503508 ]
Brock Noland edited comment on MRUNIT-161 at 11/25/12 4:26 PM:
---------------------------------------------------------------
@dbeech Do you know the difference is between the getLocal*Cache* and getCache* methods? It looks like they are using a different configuration key.
was (Author: brocknoland):
~dbeech Do you know the difference is between the getLocal*Cache* and getCache* methods? It looks like they are using a different configuration key.
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503752#comment-13503752 ]
Brock Noland commented on MRUNIT-161:
-------------------------------------
Ok cool, then the contents of that patch should work though we might consider allowing a configuration object in the driver constructors.
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated MRUNIT-161:
--------------------------------
Attachment: MRUNIT-161-0.patch
Here is a patch which implements most of the requested functionality. I do, however, wonder about the getLocal* methods because we don't seem to be setting "local" files anywhere.
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509877#comment-13509877 ]
Brock Noland commented on MRUNIT-161:
-------------------------------------
OK I'll see if I can get it compiling against hadoop 1.
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Assignee: Brock Noland
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503743#comment-13503743 ]
Dave Beech commented on MRUNIT-161:
-----------------------------------
the getLocal* methods are what the user would call in a mapper or reducer class to access the cache files that have been "localized". That's to say, copied to local disk into the task's working area by the mapreduce framework.
We do set the "local" files in mrunit - it's done in TestDriver's initDistributedCache method.
I'll take a look at the patch later - thanks Brock.
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Beech reassigned MRUNIT-161:
---------------------------------
Assignee: Brock Noland
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Assignee: Brock Noland
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509860#comment-13509860 ]
Dave Beech commented on MRUNIT-161:
-----------------------------------
Hi Brock - I applied your patch but then the code won't build against hadoop 1. o.a.h.mapreduce.TaskInputOutputContext is missing some methods, like getSymlink
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Assignee: Brock Noland
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13502791#comment-13502791 ]
Brock Noland commented on MRUNIT-161:
-------------------------------------
There are bunch of methods relating to Dcache in JobContextImpl. They look easy enough to implement, we'd just need the configuration object to do so.
public URI[] getCacheArchives() throws IOException {
return DistributedCache.getCacheArchives(conf);
}
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Priority: Blocker
> Fix For: 1.0.0
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-161) some distributed cache apis not
working - JobContext getCacheArchives()
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503508#comment-13503508 ]
Brock Noland commented on MRUNIT-161:
-------------------------------------
~dbeech Do you know the difference is between the getLocal*Cache* and getCache* methods? It looks like they are using a different configuration key.
> some distributed cache apis not working - JobContext getCacheArchives()
> -----------------------------------------------------------------------
>
> Key: MRUNIT-161
> URL: https://issues.apache.org/jira/browse/MRUNIT-161
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Patrick Hunt
> Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: MRUNIT-161-0.patch
>
>
> I'm using the new MRUnit method withCacheArchive available in 1.0.0.
> DistributedCache.getLocalCacheArchives(conf) returns a non-empty array, however
> org.apache.hadoop.mapreduce.JobContext.getCacheArchives()
> in the mapper returns null. It seems that the context mock doesn't have these methods implemented?
> I'm using hadoop2, where getLocalCacheArchives is listed as deprecated.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira