You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrunit.apache.org by Sirish Chandraa Dhananjaya <si...@gmail.com> on 2013/11/26 22:24:07 UTC
MRUnit 1.0.0 not working with Hadoop 0.20.203.2
I was originally using mrunit 0.20.1+152 which did not have the ability to
specify custom configuration parameters. So I jumped to the latest mrunit
1.0.0 for hadoop_1. I noticed that even a test case that succeeds with
mrunit 0.20.1+152 throws an internal exception on mrunit 1.0.0 for hadoop1.
Test case is as simple as setting an input, an output and calling runTest()
on the reduce driver object.
I get the following error :
java.lang.ClassCastException:
org.apache.hadoop.mrunit.internal.mapreduce.MockReduceContextWrapper$5$1
cannot be cast to org.apache.hadoop.mapreduce.ReduceContext$ValueIterator
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
at
org.apache.hadoop.mrunit.mapreduce.ReduceDriver.run(ReduceDriver.java:143)
at
org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:574)
at
org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:561)
Any help would be appreciated.
Thanks,
Sirish
Re: MRUnit 1.0.0 not working with Hadoop 0.20.203.2
Posted by Brock Noland <br...@cloudera.com>.
On Tue, Nov 26, 2013 at 8:55 PM, Sirish Chandraa Dhananjaya <
sirsa.chandraa@gmail.com> wrote:
> Hi Bertrand,
>
> A minor correction : The version of hadoop I was using was 0.20.203.0
>
> I just tried mrunit 1.0.0 with Hadoop 0.20.205.0 and that works. Going by
> the mrunit blog, it seems mrunit for hadoop 1 supports the 0.20.X line
> according to the docs. Should I go ahead and file a bug ?
>
Hmm, you used the hadoop2 jar against hadoop 0.20 and it worked?
Re: MRUnit 1.0.0 not working with Hadoop 0.20.203.2
Posted by Sirish Chandraa Dhananjaya <si...@gmail.com>.
Hi Bertrand,
A minor correction : The version of hadoop I was using was 0.20.203.0
I just tried mrunit 1.0.0 with Hadoop 0.20.205.0 and that works. Going by
the mrunit blog, it seems mrunit for hadoop 1 supports the 0.20.X line
according to the docs. Should I go ahead and file a bug ?
BTW, I am using a custom tool that takes care of dependencies.
Code Snippet (inside test class) :
ReduceDriver<Text, CustomWritable, Text, Text> reduceDriver;
@Before
public void setUp() {
Reducer<Text,CustomWritable,Text,Text> reducer = new AdditionReducer();
reduceDriver = new ReduceDriver<Text,CustomWritable,Text,Text>();
reduceDriver.setReducer(reducer);
}
@Test
public void testReducer throws IOException {
reduceDriver.withInput(new Text("K"), Arrays.asList(new
CustomWritable(1,2,3), new CustomWritable(4,5,6))
.withOutput(new Text("K"), new Text("5,7,9"))
.runTest();
}
On Tue, Nov 26, 2013 at 2:11 PM, Bertrand Dechoux <de...@gmail.com>wrote:
> I am assuming that for private reasons you can't upgrade to a version 1 of
> Hadoop even though now the version 2 is stable?
>
> It might be a change within the API. Are you using maven? What is the
> behavior of mrunit 1.0.0 when you select a real hadoop 1?
>
> Would you be able to provide a snippet?
>
> Regard
>
> Bertrand
>
>
> On Tue, Nov 26, 2013 at 10:24 PM, Sirish Chandraa Dhananjaya <
> sirsa.chandraa@gmail.com> wrote:
>
>> I was originally using mrunit 0.20.1+152 which did not have the ability
>> to specify custom configuration parameters. So I jumped to the latest
>> mrunit 1.0.0 for hadoop_1. I noticed that even a test case that succeeds
>> with mrunit 0.20.1+152 throws an internal exception on mrunit 1.0.0 for
>> hadoop1.
>>
>> Test case is as simple as setting an input, an output and calling
>> runTest() on the reduce driver object.
>>
>> I get the following error :
>>
>> java.lang.ClassCastException:
>> org.apache.hadoop.mrunit.internal.mapreduce.MockReduceContextWrapper$5$1
>> cannot be cast to org.apache.hadoop.mapreduce.ReduceContext$ValueIterator
>>
>> at
>> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
>>
>> at
>> org.apache.hadoop.mrunit.mapreduce.ReduceDriver.run(ReduceDriver.java:143)
>>
>> at
>> org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:574)
>>
>> at
>> org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:561)
>>
>>
>> Any help would be appreciated.
>>
>>
>> Thanks,
>>
>> Sirish
>>
>>
>>
>>
>
>
> --
> Bertrand Dechoux
>
Re: MRUnit 1.0.0 not working with Hadoop 0.20.203.2
Posted by Bertrand Dechoux <de...@gmail.com>.
I am assuming that for private reasons you can't upgrade to a version 1 of
Hadoop even though now the version 2 is stable?
It might be a change within the API. Are you using maven? What is the
behavior of mrunit 1.0.0 when you select a real hadoop 1?
Would you be able to provide a snippet?
Regard
Bertrand
On Tue, Nov 26, 2013 at 10:24 PM, Sirish Chandraa Dhananjaya <
sirsa.chandraa@gmail.com> wrote:
> I was originally using mrunit 0.20.1+152 which did not have the ability to
> specify custom configuration parameters. So I jumped to the latest mrunit
> 1.0.0 for hadoop_1. I noticed that even a test case that succeeds with
> mrunit 0.20.1+152 throws an internal exception on mrunit 1.0.0 for hadoop1.
>
> Test case is as simple as setting an input, an output and calling
> runTest() on the reduce driver object.
>
> I get the following error :
>
> java.lang.ClassCastException:
> org.apache.hadoop.mrunit.internal.mapreduce.MockReduceContextWrapper$5$1
> cannot be cast to org.apache.hadoop.mapreduce.ReduceContext$ValueIterator
>
> at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
>
> at
> org.apache.hadoop.mrunit.mapreduce.ReduceDriver.run(ReduceDriver.java:143)
>
> at
> org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:574)
>
> at
> org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:561)
>
>
> Any help would be appreciated.
>
>
> Thanks,
>
> Sirish
>
>
>
>
--
Bertrand Dechoux