[jira] [Commented] (MRUNIT-99) dont force K2 in MapReduceDriver's to be Comparable since Hadoop dont enforce this on the map output key and value classes

["Jim Donofrio (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MRUNIT-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254488#comment-13254488 ] 

Jim Donofrio commented on MRUNIT-99:
------------------------------------

With the release going out tomorrow I put together a quick patch. I removed extends Comparable from K2 in both MapReduceDriver classes. In MapReduceDriverBase if keyGroupComparator is null then I use new JobConf(getConfiguration()).getOutputValueGroupingComparator()  which if the group comparator isnt set in the conf uses the key comparator and if that isnt set uses WritableComparator.get

I think when designing the new api we can still have these convenience methods, withMapper, etc although we could get rid of them but we should use the conf to store all the settings to mimic hadoop. We could just have a getConf() method and allow users to set the necessary arguments in there. Then when we test the job we could throw a specific exception if they did not remember to set a property that we need in the conf

committed in 1326461 in 0.9.0 and 1326463 in trunk
                
> dont force K2 in MapReduceDriver's to be Comparable since Hadoop dont enforce this on the map output key and value classes
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MRUNIT-99
>                 URL: https://issues.apache.org/jira/browse/MRUNIT-99
>             Project: MRUnit
>          Issue Type: Bug
>    Affects Versions: 0.8.1
>            Reporter: Jim Donofrio
>            Assignee: Jim Donofrio
>             Fix For: 0.9.0
>
>
> We should not force K2 in MapReduceDriver's to be Comparable because Hadoop does not enforce this on the map output key and value classes. This makes MRUnit difficult to use when using non Writable classes that dont need to be Comparable but define their own key comparators in the JobConf
> I realized this after seeing [1] on stackoverflow. I am still trying to investigate if we have other conflicts with Avro
> [1]: http://stackoverflow.com/questions/10119616/use-mrunit-and-avro-together

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira