You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrunit.apache.org by Mike Bernico <mi...@gmail.com> on 2013/07/25 00:25:59 UTC

Help testing a reducer with MRUnit 1.0.0 Hadoop2

Hi Everyone,

I'm experiencing some odd behavior, that's likely my fault, with MRUnit.
 Would someone mind taking a look?   It seems like no matter what I do the
reducer test harness is just echoing back whatever I send to the reducer.
I made a really simple/dumb recreate of the issue here.

Many thanks in advance...


*import* java.io.IOException;
*import* org.apache.hadoop.mapreduce.Reducer;
*import* org.apache.hadoop.io.IntWritable;

*public* *class* TestReducer
                *extends*
                Reducer<IntWritable, IntWritable, IntWritable, IntWritable>
{

        *public* *void* Reduce(IntWritable key,
                        Iterable<IntWritable> values, Context context)
                        *throws* IOException, InterruptedException {

                *int* *i*;
                *for* (IntWritable value : values) {
                        i = value.get();
                }
                context.write(*new* IntWritable(7), *new* IntWritable(11));

        }
}

*import* java.io.IOException;
*import* java.util.ArrayList;
*import* java.util.List;

*import* org.apache.hadoop.io.IntWritable;
*import* org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
*import* org.junit.Test;



*public* *class* TestReducerTest {

        @Test
        *public* *void* testReducer()
                        *throws* IOException, InterruptedException {

                List<IntWritable> theList = *new* ArrayList <IntWritable>();
                theList.add(*new* IntWritable(5));

                *new*
 ReduceDriver<IntWritable,IntWritable,IntWritable,IntWritable>()
                        .withReducer(*new* TestReducer())
                        .withInput(*new* IntWritable(1), theList)
                        .withOutput(*new* IntWritable(7), *new*
 IntWritable(11))
                        .runTest();

        }

}

13/07/24 17:16:13 ERROR mrunit.TestDriver: Missing expected output (7, 11)
at position 0.
13/07/24 17:16:13 ERROR mrunit.TestDriver: Received unexpected output (1,
5) at position 0.

Re: Help testing a reducer with MRUnit 1.0.0 Hadoop2

Posted by Bertrand Dechoux <de...@gmail.com>.
And that's why @Override should always be used (especially with Hadoop).

http://docs.oracle.com/javase/7/docs/api/java/lang/Override.html

BTW you can also simplify your code :
* use ReduceDriver.newReduceDriver(new TestReducer())
* instead of ** *new*
 ReduceDriver<IntWritable,IntWritable,IntWritable,IntWritable>()
                        .withReducer(*new* TestReducer())

Bertrand


On Thu, Jul 25, 2013 at 1:43 AM, Mike Bernico <mi...@gmail.com>wrote:

> Oops, yes that was it.   I must have stared at that for an hour too.
>
> Thanks!
>
>
>
>
> On Wed, Jul 24, 2013 at 5:30 PM, Sudhin Varghese <su...@gmail.com> wrote:
>
>> Since your reduce method is overridden incorrectly as Reduce, I assume
>> its calling the default identity function.
>>
>> Thanks,
>> _sudhin_
>> http://www.sudhcha.com
>>
>> On Jul 24, 2013, at 6:25 PM, Mike Bernico <mi...@gmail.com> wrote:
>>
>> > Reduce
>>
>
>

Re: Help testing a reducer with MRUnit 1.0.0 Hadoop2

Posted by Mike Bernico <mi...@gmail.com>.
Oops, yes that was it.   I must have stared at that for an hour too.

Thanks!




On Wed, Jul 24, 2013 at 5:30 PM, Sudhin Varghese <su...@gmail.com> wrote:

> Since your reduce method is overridden incorrectly as Reduce, I assume its
> calling the default identity function.
>
> Thanks,
> _sudhin_
> http://www.sudhcha.com
>
> On Jul 24, 2013, at 6:25 PM, Mike Bernico <mi...@gmail.com> wrote:
>
> > Reduce
>

Re: Help testing a reducer with MRUnit 1.0.0 Hadoop2

Posted by Sudhin Varghese <su...@gmail.com>.
Since your reduce method is overridden incorrectly as Reduce, I assume its calling the default identity function.

Thanks,
_sudhin_
http://www.sudhcha.com

On Jul 24, 2013, at 6:25 PM, Mike Bernico <mi...@gmail.com> wrote:

> Reduce