You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mrunit.apache.org by "Jim Donofrio (Created) (JIRA)" <ji...@apache.org> on 2012/04/16 04:01:16 UTC
[jira] [Created] (MRUNIT-99) dont force K2 in MapReduceDriver's to
be Comparable since Hadoop dont enforce this on the map output key and
value classes
dont force K2 in MapReduceDriver's to be Comparable since Hadoop dont enforce this on the map output key and value classes
--------------------------------------------------------------------------------------------------------------------------
Key: MRUNIT-99
URL: https://issues.apache.org/jira/browse/MRUNIT-99
Project: MRUnit
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Jim Donofrio
Assignee: Jim Donofrio
Fix For: 0.9.0
We should not force K2 in MapReduceDriver's to be Comparable because Hadoop does not enforce this on the map output key and value classes. This makes MRUnit difficult to use when using non Writable classes that dont need to be Comparable but define their own key comparators in the JobConf
I realized this after seeing [1] on stackoverflow. I am still trying to investigate if we have other conflicts with Avro
[1]: http://stackoverflow.com/questions/10119616/use-mrunit-and-avro-together
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MRUNIT-99) dont force K2 in MapReduceDriver's to
be Comparable since Hadoop dont enforce this on the map output key and
value classes
Posted by "Jim Donofrio (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-99?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Donofrio resolved MRUNIT-99.
--------------------------------
Resolution: Fixed
> dont force K2 in MapReduceDriver's to be Comparable since Hadoop dont enforce this on the map output key and value classes
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-99
> URL: https://issues.apache.org/jira/browse/MRUNIT-99
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 0.9.0
>
>
> We should not force K2 in MapReduceDriver's to be Comparable because Hadoop does not enforce this on the map output key and value classes. This makes MRUnit difficult to use when using non Writable classes that dont need to be Comparable but define their own key comparators in the JobConf
> I realized this after seeing [1] on stackoverflow. I am still trying to investigate if we have other conflicts with Avro
> [1]: http://stackoverflow.com/questions/10119616/use-mrunit-and-avro-together
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MRUNIT-99) dont force K2 in MapReduceDriver's
to be Comparable since Hadoop dont enforce this on the map output key and
value classes
Posted by "Jim Donofrio (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MRUNIT-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254488#comment-13254488 ]
Jim Donofrio commented on MRUNIT-99:
------------------------------------
With the release going out tomorrow I put together a quick patch. I removed extends Comparable from K2 in both MapReduceDriver classes. In MapReduceDriverBase if keyGroupComparator is null then I use new JobConf(getConfiguration()).getOutputValueGroupingComparator() which if the group comparator isnt set in the conf uses the key comparator and if that isnt set uses WritableComparator.get
I think when designing the new api we can still have these convenience methods, withMapper, etc although we could get rid of them but we should use the conf to store all the settings to mimic hadoop. We could just have a getConf() method and allow users to set the necessary arguments in there. Then when we test the job we could throw a specific exception if they did not remember to set a property that we need in the conf
committed in 1326461 in 0.9.0 and 1326463 in trunk
> dont force K2 in MapReduceDriver's to be Comparable since Hadoop dont enforce this on the map output key and value classes
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: MRUNIT-99
> URL: https://issues.apache.org/jira/browse/MRUNIT-99
> Project: MRUnit
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Jim Donofrio
> Assignee: Jim Donofrio
> Fix For: 0.9.0
>
>
> We should not force K2 in MapReduceDriver's to be Comparable because Hadoop does not enforce this on the map output key and value classes. This makes MRUnit difficult to use when using non Writable classes that dont need to be Comparable but define their own key comparators in the JobConf
> I realized this after seeing [1] on stackoverflow. I am still trying to investigate if we have other conflicts with Avro
> [1]: http://stackoverflow.com/questions/10119616/use-mrunit-and-avro-together
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira