You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrunit.apache.org by Mike Bernico <mi...@gmail.com> on 2013/07/25 00:25:59 UTC
Help testing a reducer with MRUnit 1.0.0 Hadoop2
Hi Everyone,
I'm experiencing some odd behavior, that's likely my fault, with MRUnit.
Would someone mind taking a look? It seems like no matter what I do the
reducer test harness is just echoing back whatever I send to the reducer.
I made a really simple/dumb recreate of the issue here.
Many thanks in advance...
*import* java.io.IOException;
*import* org.apache.hadoop.mapreduce.Reducer;
*import* org.apache.hadoop.io.IntWritable;
*public* *class* TestReducer
*extends*
Reducer<IntWritable, IntWritable, IntWritable, IntWritable>
{
*public* *void* Reduce(IntWritable key,
Iterable<IntWritable> values, Context context)
*throws* IOException, InterruptedException {
*int* *i*;
*for* (IntWritable value : values) {
i = value.get();
}
context.write(*new* IntWritable(7), *new* IntWritable(11));
}
}
*import* java.io.IOException;
*import* java.util.ArrayList;
*import* java.util.List;
*import* org.apache.hadoop.io.IntWritable;
*import* org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
*import* org.junit.Test;
*public* *class* TestReducerTest {
@Test
*public* *void* testReducer()
*throws* IOException, InterruptedException {
List<IntWritable> theList = *new* ArrayList <IntWritable>();
theList.add(*new* IntWritable(5));
*new*
ReduceDriver<IntWritable,IntWritable,IntWritable,IntWritable>()
.withReducer(*new* TestReducer())
.withInput(*new* IntWritable(1), theList)
.withOutput(*new* IntWritable(7), *new*
IntWritable(11))
.runTest();
}
}
13/07/24 17:16:13 ERROR mrunit.TestDriver: Missing expected output (7, 11)
at position 0.
13/07/24 17:16:13 ERROR mrunit.TestDriver: Received unexpected output (1,
5) at position 0.
Re: Help testing a reducer with MRUnit 1.0.0 Hadoop2
Posted by Bertrand Dechoux <de...@gmail.com>.
And that's why @Override should always be used (especially with Hadoop).
http://docs.oracle.com/javase/7/docs/api/java/lang/Override.html
BTW you can also simplify your code :
* use ReduceDriver.newReduceDriver(new TestReducer())
* instead of ** *new*
ReduceDriver<IntWritable,IntWritable,IntWritable,IntWritable>()
.withReducer(*new* TestReducer())
Bertrand
On Thu, Jul 25, 2013 at 1:43 AM, Mike Bernico <mi...@gmail.com>wrote:
> Oops, yes that was it. I must have stared at that for an hour too.
>
> Thanks!
>
>
>
>
> On Wed, Jul 24, 2013 at 5:30 PM, Sudhin Varghese <su...@gmail.com> wrote:
>
>> Since your reduce method is overridden incorrectly as Reduce, I assume
>> its calling the default identity function.
>>
>> Thanks,
>> _sudhin_
>> http://www.sudhcha.com
>>
>> On Jul 24, 2013, at 6:25 PM, Mike Bernico <mi...@gmail.com> wrote:
>>
>> > Reduce
>>
>
>
Re: Help testing a reducer with MRUnit 1.0.0 Hadoop2
Posted by Mike Bernico <mi...@gmail.com>.
Oops, yes that was it. I must have stared at that for an hour too.
Thanks!
On Wed, Jul 24, 2013 at 5:30 PM, Sudhin Varghese <su...@gmail.com> wrote:
> Since your reduce method is overridden incorrectly as Reduce, I assume its
> calling the default identity function.
>
> Thanks,
> _sudhin_
> http://www.sudhcha.com
>
> On Jul 24, 2013, at 6:25 PM, Mike Bernico <mi...@gmail.com> wrote:
>
> > Reduce
>
Re: Help testing a reducer with MRUnit 1.0.0 Hadoop2
Posted by Sudhin Varghese <su...@gmail.com>.
Since your reduce method is overridden incorrectly as Reduce, I assume its calling the default identity function.
Thanks,
_sudhin_
http://www.sudhcha.com
On Jul 24, 2013, at 6:25 PM, Mike Bernico <mi...@gmail.com> wrote:
> Reduce