You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Brock Noland (Created) (JIRA)" <ji...@apache.org> on 2012/04/13 23:15:17 UTC
[jira] [Created] (MRUNIT-98) DistributedCache support
DistributedCache support
------------------------
Key: MRUNIT-98
URL: https://issues.apache.org/jira/browse/MRUNIT-98
Project: MRUnit
Issue Type: Bug
Reporter: Brock Noland
Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-98) DistributedCache support
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270363#comment-13270363 ]
Jim Donofrio commented on MRUNIT-98:
------------------------------------
Yes I am finishing up MRUNIT-101 and originally I had planned to extend FileSystem with a MemoryFileSystem class. Brock and Joey Echeverria convinced me otherwise in MRUNIT-102. It was much easier to just create random directories in temp. I am not too familar with these mock frameworks but would there be a way to make File even work with any filesystem besides the local one?
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-98) DistributedCache support
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio updated MRUNIT-98:
-------------------------------
Fix Version/s: 1.0.0
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Brock Noland
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-98) DistributedCache support
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270214#comment-13270214 ]
Jim Donofrio commented on MRUNIT-98:
------------------------------------
Yeah symlinks would be a mess because we would create a bunch of shortcuts in the current directory which in eclipse would be the root of their project
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-98) DistributedCache support
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270387#comment-13270387 ]
Jim Donofrio commented on MRUNIT-98:
------------------------------------
Yes I agree that our goal should always be to run the same code that will run on the cluster, otherwise we are not really testing anything, we are testing test code.
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-98) DistributedCache support
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270356#comment-13270356 ]
Dave Beech commented on MRUNIT-98:
----------------------------------
My first plan was to try and simulate the cache without reading/writing any actual files, by mocking the calls to DistributedCache and backing it with data preloaded into memory by the test. Disk I/O in unit tests seems wrong, I always try and avoid it wherever possible! But, I'm probably just over-complicating things - Jim, your suggestion might be the way forward for a first stab at this. I'll try and put something together later and get it over to you guys to check out.
Agree the symlinks thing would be a mess, and would also cause a problem for Windows users. It's annoying as that's the method I prefer using to get files from the cache myself.
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-98) DistributedCache support
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio updated MRUNIT-98:
-------------------------------
Reporter: Ajay Srivastava (was: Brock Noland)
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-98) DistributedCache support
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270204#comment-13270204 ]
Brock Noland commented on MRUNIT-98:
------------------------------------
Some people use fact that in hadoop you have symlink to the file in the tasks CWD (I think Tom White even says this one edition of his book) and do the following in the configure method:
File file = new File("relative-path-to-file");
I am not sure how we would support that without creating symlinks in the jvms CWD which could be ugly if not cleaned up. Maybe we don't need to support that use.
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-98) DistributedCache support
Posted by "Dave Beech (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270379#comment-13270379 ]
Dave Beech commented on MRUNIT-98:
----------------------------------
Well, it wouldn't be the File object you'd need to mock as such, it'd be whatever reader or stream you open to consume that file.
Having thought about it some more, it's probably not worth the effort. I'm sure it's possible, but the user's mapper or configure method code would need to be written in a way which allows the mock objects to be injected in the right places, and would require a lot of non-mrunit test fixture setup code that I doubt anyone would want to write.
Using temp files would definitely be a lot easier. Let's go with that.
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-98) DistributedCache support
Posted by "Jim Donofrio (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13270175#comment-13270175 ]
Jim Donofrio commented on MRUNIT-98:
------------------------------------
Dave,
I saw you commented on this MRUNIT-97, any thoughts on this?
I was thinking all new really need to do is copy some jobconf parameters and files around. We could just generate random dir's in the temp dir such as:
File outputPath = new File(System.getProperty("java.io.tmpdir"), "mrunit-" + Math.random());
and then copy the files there and unzip them if necessary and then set the destination conf parameter
Thoughts?
> DistributedCache support
> ------------------------
>
> Key: MRUNIT-98
> URL: https://issues.apache.org/jira/browse/MRUNIT-98
> Project: MRUnit
> Issue Type: Bug
> Reporter: Ajay Srivastava
> Fix For: 1.0.0
>
>
> Created as MRUNIT-97 became a patch for InputSplits.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira