You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Matthew Rathbone (Created) (JIRA)" <ji...@apache.org> on 2012/03/14 16:54:42 UTC
[jira] [Created] (MRUNIT-88) MRUnit should support custom
partitioners, comparator, and groupComparator
MRUnit should support custom partitioners, comparator, and groupComparator
--------------------------------------------------------------------------
Key: MRUNIT-88
URL: https://issues.apache.org/jira/browse/MRUNIT-88
Project: MRUnit
Issue Type: Improvement
Reporter: Matthew Rathbone
We're building something that essentially does a secondary sort, to test that we need to be able to specify comparators and partitioners.
Example:
the following two tuple keys: (id1, source1), (id1, source2)
should be grouped together based on the first value of the tuple, and their records should end up in the same reducer
To do this we have our own custom partitioner / comparator, this is what we need to test through the whole pipeline in this way:
MapReduceDriver.setPartitioner(p)
MapReduceDriver.setGroupComparator(c)
I'm not familiar enough with the MRUnit code to add this easily, but I suspect it would be pretty quick to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-88) MRUnit should support custom
partitioners, comparator, and groupComparator
Posted by "Brock Noland (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229282#comment-13229282 ]
Brock Noland commented on MRUNIT-88:
------------------------------------
You can test the comparator and grouping comparator today with MRUnit:
See KeyGrouping and KeyOrdering comparators:
http://incubator.apache.org/mrunit/apidocs/org/apache/hadoop/mrunit/MapReduceDriver.html
The partitioner cannot be tested today. Over in MRUNIT-69 Jim is working to provide a the ability test more things, but I am not sure we will get to the point of testing the Partitioner in the case of more than 1 reducer because Hadoop's local job runner only allows a single reducer.
> MRUnit should support custom partitioners, comparator, and groupComparator
> --------------------------------------------------------------------------
>
> Key: MRUNIT-88
> URL: https://issues.apache.org/jira/browse/MRUNIT-88
> Project: MRUnit
> Issue Type: Improvement
> Reporter: Matthew Rathbone
> Labels: partitioners
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> We're building something that essentially does a secondary sort, to test that we need to be able to specify comparators and partitioners.
> Example:
> the following two tuple keys: (id1, source1), (id1, source2)
> should be grouped together based on the first value of the tuple, and their records should end up in the same reducer
> To do this we have our own custom partitioner / comparator, this is what we need to test through the whole pipeline in this way:
> MapReduceDriver.setPartitioner(p)
> MapReduceDriver.setGroupComparator(c)
> I'm not familiar enough with the MRUnit code to add this easily, but I suspect it would be pretty quick to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-88) MRUnit should support custom
partitioners, comparator, and groupComparator
Posted by "Jim Donofrio (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-88?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229844#comment-13229844 ]
Jim Donofrio commented on MRUNIT-88:
------------------------------------
Yes testing a Partitioner would be tough because the localjobrunner as Brock said forces 1 reducer. If we integrated the minimrcluster class we could have multiple reducers. Although there is a hadoop JIRA to add support for multiple reducers in localjobrunner: MAPREDUCE-434
> MRUnit should support custom partitioners, comparator, and groupComparator
> --------------------------------------------------------------------------
>
> Key: MRUNIT-88
> URL: https://issues.apache.org/jira/browse/MRUNIT-88
> Project: MRUnit
> Issue Type: Improvement
> Reporter: Matthew Rathbone
> Labels: partitioners
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> We're building something that essentially does a secondary sort, to test that we need to be able to specify comparators and partitioners.
> Example:
> the following two tuple keys: (id1, source1), (id1, source2)
> should be grouped together based on the first value of the tuple, and their records should end up in the same reducer
> To do this we have our own custom partitioner / comparator, this is what we need to test through the whole pipeline in this way:
> MapReduceDriver.setPartitioner(p)
> MapReduceDriver.setGroupComparator(c)
> I'm not familiar enough with the MRUnit code to add this easily, but I suspect it would be pretty quick to do.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira