You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Ajay Srivastava (Created) (JIRA)" <ji...@apache.org> on 2012/04/11 15:23:16 UTC
[jira] [Created] (MRUNIT-97) Can't use MRunit if Context and
DistributedCache class is used in mapreduce job
Can't use MRunit if Context and DistributedCache class is used in mapreduce job
-------------------------------------------------------------------------------
Key: MRUNIT-97
URL: https://issues.apache.org/jira/browse/MRUNIT-97
Project: MRUnit
Issue Type: Bug
Affects Versions: 0.8.0
Reporter: Ajay Srivastava
I believe that you can't test input split filename and distributed cache at the moment.
If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
Jarcec
On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
Hi,
I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
For an example, in one of the map reduce job I have following code in setup method of Mapper class.
context.getInputSplit()).getPath();
I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
Regards,
Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-97) InputSplit causes NullPointerException
in mapreduce api
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated MRUNIT-97:
-------------------------------
Fix Version/s: (was: 1.0.0)
0.9.0
> InputSplit causes NullPointerException in mapreduce api
> -------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Assignee: Dave Beech
> Fix For: 0.9.0
>
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-97) InputSplit causes NullPointerException
in mapreduce api
Posted by "Brock Noland (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated MRUNIT-97:
-------------------------------
Summary: InputSplit causes NullPointerException in mapreduce api (was: Can't use MRunit if Context and DistributedCache class is used in mapreduce job)
> InputSplit causes NullPointerException in mapreduce api
> -------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MRUNIT-97) InputSplit causes
NullPointerException in mapreduce api
Posted by "Brock Noland (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland resolved MRUNIT-97.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.0.0
Thank you for the patch Dave! Committed in 1325959.
Ajay, I created a seperate JIRA MRUNIT-98 for DCache Support
> InputSplit causes NullPointerException in mapreduce api
> -------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Assignee: Dave Beech
> Fix For: 1.0.0
>
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MRUNIT-97) Can't use MRunit if Context and
DistributedCache class is used in mapreduce job
Posted by "Dave Beech (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dave Beech updated MRUNIT-97:
-----------------------------
Attachment: MRUNIT-97.patch
Hi. I've created a patch which addresses the NullPointerException problem - it simply adds an extra stubbed method call on MockMapContextWrapper which returns an instance of MockInputSplit when getInputSplit() is called.
This could be further extended - maybe you'd want to specify in your test what the file path / length of the input split should be? At the moment these values are hardcoded but this could easily be changed.
As for testing DistributedCache, I think this would be more difficult since these are mostly static methods and cannot be mocked by Mockito. However, I believe there are other mocking frameworks that can do this (PowerMock for instance). Has anyone else got experience here?
> Can't use MRunit if Context and DistributedCache class is used in mapreduce job
> -------------------------------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-97) InputSplit causes
NullPointerException in mapreduce api
Posted by "Dave Beech (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254053#comment-13254053 ]
Dave Beech commented on MRUNIT-97:
----------------------------------
Thanks Brock! I'll take a look at MRUNIT-98 next
> InputSplit causes NullPointerException in mapreduce api
> -------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Assignee: Dave Beech
> Fix For: 1.0.0
>
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MRUNIT-97) InputSplit causes
NullPointerException in mapreduce api
Posted by "Brock Noland (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland reassigned MRUNIT-97:
----------------------------------
Assignee: Dave Beech
> InputSplit causes NullPointerException in mapreduce api
> -------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Assignee: Dave Beech
> Fix For: 1.0.0
>
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-97) InputSplit causes
NullPointerException in mapreduce api
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254423#comment-13254423 ]
Brock Noland commented on MRUNIT-97:
------------------------------------
1326419. in the 0.9.0 branch
> InputSplit causes NullPointerException in mapreduce api
> -------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Assignee: Dave Beech
> Fix For: 0.9.0
>
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-97) InputSplit causes
NullPointerException in mapreduce api
Posted by "Ajay Srivastava (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254514#comment-13254514 ]
Ajay Srivastava commented on MRUNIT-97:
---------------------------------------
Thanks Brock and Dave.
I will try the patch early next week.
> InputSplit causes NullPointerException in mapreduce api
> -------------------------------------------------------
>
> Key: MRUNIT-97
> URL: https://issues.apache.org/jira/browse/MRUNIT-97
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.0
> Reporter: Ajay Srivastava
> Assignee: Dave Beech
> Fix For: 0.9.0
>
> Attachments: MRUNIT-97.patch
>
>
> I believe that you can't test input split filename and distributed cache at the moment.
> If you could file a JIRA on https://issues.apache.org/jira/browse/MRUNIT, I'm sure that someone would look on that sooner or later.
> Jarcec
> On Apr 11, 2012, at 7:53 AM, Ajay Srivastava wrote:
> Hi,
> I am planning to use mrunit in one of the projects and written some test cases. These test cases run fine for simple map reduce jobs but give error for most of the others.
> For an example, in one of the map reduce job I have following code in setup method of Mapper class.
> context.getInputSplit()).getPath();
> I get null pointer exception at above statement. Is there a way that I can set this context in test code ? If not, how can this kind of mrjobs be tested.
> Similarly we are using DistributedCache in some of our mr jobs. How can I test such jobs? Is there a mockup available for DistributedCache class in mrunit.
> Regards,
> Ajay Srivastava
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira