You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2009/03/17 21:37:50 UTC
[jira] Created: (HADOOP-5518) MRUnit unit test library
MRUnit unit test library
------------------------
Key: HADOOP-5518
URL: https://issues.apache.org/jira/browse/HADOOP-5518
Project: Hadoop Core
Issue Type: New Feature
Reporter: Aaron Kimball
MRUnit is a tool to help authors of MapReduce programs write unit tests.
Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-5518:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Aaron!
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693972#action_12693972 ]
Hadoop QA commented on HADOOP-5518:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12404186/HADOOP-5518-2.patch
against trunk revision 760098.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 23 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
-1 release audit. The applied patch generated 662 release audit warnings (more than the trunk's current 660 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/81/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/81/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/81/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/81/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/81/console
This message is automatically generated.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Attachment: mrunit.patch
overview.html
Initial upload of MRUnit source
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Attachments: mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694123#action_12694123 ]
Steve Loughran commented on HADOOP-5518:
----------------------------------------
I am getting ahead of myself, but I'm thinking we should plan to have a redistributable hadoop-test-tools package into which allthis stuff can go. Your code would be the first release.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Status: Open (was: Patch Available)
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Attachment: (was: HADOOP-5518-branch18.patch)
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, HADOOP-5518-branch18.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693850#action_12693850 ]
Aaron Kimball commented on HADOOP-5518:
---------------------------------------
Steve, thanks for your comments. I'll fix the issue re. ivy deps. If you've got other testing tools, it might be good to have a single hadoop-0.x.y-debugging.jar or something which people could use as a single source for these sorts of things. But that's getting a bit ahead of ourselves. Can you say more about what your JUnit stuff does, and/or file a separate JIRA ticket for it?
Regarding Hudson:
* The contrib test failures were in the capacity scheduler; these are unrelated to this patch.
* I can't read the release audit warnings; that link is 404.
* Can someone please explain what the Eclipse classpath issue is, and how to fix it? Is that a result of depending on JUnit 4.5 instead of 3.8.1?
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Thomas Sandholm (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696576#action_12696576 ]
Thomas Sandholm commented on HADOOP-5518:
-----------------------------------------
Seem like this commit broke the automated test-patch script. The eclipse classpath check always fails (gives -1 as seen above for all patches).
The error can be reproduced by doing a diff on:
find build/ivy/lib/Hadoop/common/ lib/ src/test/lib/ -name '*.jar' |sort
and
sed -n 's@.*kind="lib".*path="\(.*jar\)".*@\1@p' < .eclipse.templates/.classpath | sort
I presume you just need to add the new junit version to the eclipse file 'eclipse.templates/.classpath'. But I am not an eclipse user so I am not sure, I am just trying to submit a completely unrelated patch.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Status: Patch Available (was: Open)
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Status: Patch Available (was: Open)
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Assignee: Aaron Kimball
Status: Patch Available (was: Open)
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696671#action_12696671 ]
Hudson commented on HADOOP-5518:
--------------------------------
Integrated in Hadoop-trunk #800 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/800/])
. Add contrib/mrunit, a MapReduce unit test framework. Contributed by Aaron Kimball.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12696637#action_12696637 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5518:
------------------------------------------------
Thomas, you are right. We need to update eclipse classpath. Filed HADOOP-5637.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Attachment: HADOOP-5518-branch18.patch
Previous version was not the correct file; this is the correct patch.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, HADOOP-5518-branch18.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Attachment: HADOOP-5518-2.patch
Addresses Steve L.'s comment re. ivy dependencies
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Fwd: [jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by Nigel Daley <nd...@yahoo-inc.com>.
Doug, I believe you committed this patch even though it got a -1 on
Eclipse classpath. Now every patch is getting a -1. Can you please
revert or fix this?
The classpath in trunk/.eclipse.templates/.classpath has to match the
jar files in the lib directories throughout the project.
Here's a couple of the key lines from the test-patch.sh:
export DECLARED_JARS=$(sed -n 's@.*kind="lib".*path="\(.*jar\)".*@
\1@p' < .eclipse.templates/.classpath)
export PRESENT_JARS=$(find build/ivy/lib/Hadoop/common/ lib/ src/
test/lib/ -name '*.jar' |sort)
Thanks,
Nige
Begin forwarded message:
> From: "Hadoop QA (JIRA)" <ji...@apache.org>
> Date: March 31, 2009 2:54:50 PM PDT
> To: <co...@hadoop.apache.org>
> Subject: [jira] Commented: (HADOOP-5518) MRUnit unit test library
> Reply-To: core-dev@hadoop.apache.org
>
>
> [ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694297
> #action_12694297 ]
>
> Hadoop QA commented on HADOOP-5518:
> -----------------------------------
>
> -1 overall. Here are the results of testing the latest attachment
> http://issues.apache.org/jira/secure/attachment/12404202/HADOOP-5518-3.patch
> against trunk revision 760502.
>
> +1 @author. The patch does not contain any @author tags.
>
> +1 tests included. The patch appears to include 23 new or
> modified tests.
>
> +1 javadoc. The javadoc tool did not generate any warning
> messages.
>
> +1 javac. The applied patch does not increase the total number
> of javac compiler warnings.
>
> +1 findbugs. The patch does not introduce any new Findbugs
> warnings.
>
> -1 Eclipse classpath. The patch causes the Eclipse classpath to
> differ from the contents of the lib directories.
>
> +1 release audit. The applied patch does not increase the total
> number of release audit warnings.
>
> +1 core tests. The patch passed core unit tests.
>
> -1 contrib tests. The patch failed contrib unit tests.
>
> Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/testReport/
> Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
> Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/artifact/trunk/build/test/checkstyle-errors.html
> Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/console
>
> This message is automatically generated.
>
>> MRUnit unit test library
>> ------------------------
>>
>> Key: HADOOP-5518
>> URL: https://issues.apache.org/jira/browse/HADOOP-5518
>> Project: Hadoop Core
>> Issue Type: New Feature
>> Reporter: Aaron Kimball
>> Assignee: Aaron Kimball
>> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch,
>> mrunit.patch, overview.html
>>
>>
>> MRUnit is a tool to help authors of MapReduce programs write unit
>> tests.
>> Testing map() and reduce() methods requires some repeated work to
>> mock the inputs and outputs of a Mapper or Reducer class, and
>> ensure that the correct values are emitted to the OutputCollector
>> based on inputs. Also, testing a mapper and reducer together
>> requires running them with the sorted ordering guarantees made by
>> the shuffle process.
>> This library provides the above functionality to authors of maps
>> and reduces; it allows you to test maps, reduces, and map-reduce
>> pairs without needing to perform all the setup and teardown work
>> associated with running a job.
>> I believe this tool may be useful to the broader Hadoop community,
>> so I have cleaned it up and would like to see it become a "contrib"
>> module. My current environment is based on Hadoop 0.18, so this is
>> the format it expects to use. It does not have support for the new
>> Context-based interfaces for mappers/reducers.
>> I have attached the overview.html file for its javadoc, which
>> provides more synopsis and an example of usage; I am also providing
>> the current source code so that you can evaluate its structure.
>> Ideally with some feedback from the community this will move toward
>> supporting the current trunk interface soon.
>> This currently works with JUnit 4; the supplied patch changes Ivy's
>> libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901
>> as a dependency for this reason.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694297#action_12694297 ]
Hadoop QA commented on HADOOP-5518:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12404202/HADOOP-5518-3.patch
against trunk revision 760502.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 23 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/84/console
This message is automatically generated.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Status: Open (was: Patch Available)
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693412#action_12693412 ]
Hadoop QA commented on HADOOP-5518:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402415/mrunit.patch
against trunk revision 759398.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 23 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 Eclipse classpath. The patch causes the Eclipse classpath to differ from the contents of the lib directories.
-1 release audit. The applied patch generated 661 release audit warnings (more than the trunk's current 659 warnings).
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/71/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/71/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/71/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/71/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/71/console
This message is automatically generated.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5518) MRUnit unit test library
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12693808#action_12693808 ]
Steve Loughran commented on HADOOP-5518:
----------------------------------------
# this could work well with other testing improvments, though it would be preferable to be in sync with trunk
# the commons-logging dependency in ivy should be for master and not default to avoid pulling in random dependency clutter
# I have some stuff to run JUnit as an MR job; I wonder if we could integrate the stuff into the same redistributable JAR
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Attachment: HADOOP-5518-branch18.patch
Here is an 18-branch patch for this that works with 0.18.3.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, HADOOP-5518-branch18.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5518) MRUnit unit test library
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5518:
----------------------------------
Attachment: HADOOP-5518-3.patch
Patch (#3) to fix release audit warnings.
> MRUnit unit test library
> ------------------------
>
> Key: HADOOP-5518
> URL: https://issues.apache.org/jira/browse/HADOOP-5518
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5518-2.patch, HADOOP-5518-3.patch, mrunit.patch, overview.html
>
>
> MRUnit is a tool to help authors of MapReduce programs write unit tests.
> Testing map() and reduce() methods requires some repeated work to mock the inputs and outputs of a Mapper or Reducer class, and ensure that the correct values are emitted to the OutputCollector based on inputs. Also, testing a mapper and reducer together requires running them with the sorted ordering guarantees made by the shuffle process.
> This library provides the above functionality to authors of maps and reduces; it allows you to test maps, reduces, and map-reduce pairs without needing to perform all the setup and teardown work associated with running a job.
> I believe this tool may be useful to the broader Hadoop community, so I have cleaned it up and would like to see it become a "contrib" module. My current environment is based on Hadoop 0.18, so this is the format it expects to use. It does not have support for the new Context-based interfaces for mappers/reducers.
> I have attached the overview.html file for its javadoc, which provides more synopsis and an example of usage; I am also providing the current source code so that you can evaluate its structure.
> Ideally with some feedback from the community this will move toward supporting the current trunk interface soon.
> This currently works with JUnit 4; the supplied patch changes Ivy's libraries.properties file to use JUnit 4.5. I'm marking HADOOP-4901 as a dependency for this reason.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.