You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Varene Olivier <va...@echo.fr> on 2010/05/27 10:40:36 UTC
Mapper Reducer : Unit Test and mocking with static variables
Hello to all,
first of all many thanks for this great piece of software you are all
contributing to :)
I am actually creating a program in Hadoop MapReduce, but I intend to
massively use JUnit Tests to validate my code.
To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
function with it, but inside my Mapper.map function, I am using Counters
to maintain internal statistics. The update of those counters make the
test fails with a null pointer (JavaNullPointerException).
There is an issue "Counters" of variable visibility from my test ...
Do you have any idea how to mock/validate/overcome this issue ?
Thanks a lots
Best Regards
Olivier Varene
: Code :
--------
private static enum Counters {NBLINES};
private static IntWritable one = new IntWritable(1);
private static LongWritable oneL = new LongWritable(1);
// Mapper
/**
* outputs 1 for each line encountered
*/
public static class LCMapper extends
Mapper<LongWritable, Text, IntWritable, LongWritable>
{
private final static IntWritable one = new IntWritable(1);
private final static LongWritable oneL = new LongWritable(1);
/**
* for each line encountered outputs 1 for the key 1
*/
public void map(LongWritable p_key, Text p_inputs, Context p_context)
throws IOException, InterruptedException
{
p_context.getCounter(Counters.NBLINES).increment(1);
p_context.write(one,oneL);
} // end map
} // end Mapper
: Test :
--------
@Test
public void testLCMapper() throws IOException, InterruptedException
{
LCMapper mapper = new LCMapper();
Text value = new Text("ligneA\nlignB\n");
// mock the map job execution context
Context context = mock(Context.class);
try {
mapper.map(null, value, context);
} catch (NullPointerException e)
{
// exception raised by context.getCounter(...) null pointer
// HOW TO MOCK ???
// Counter is a static enum from the englobing class ...
// issue with variable visibility (from map function) in this test
}
// verify that the Mapper issued (one,oneL) pair
verify(context).write(one,oneL);
} // end testLCMapper
Re: Mapper Reducer : Unit Test and mocking with static variables
Posted by Varene Olivier <va...@echo.fr>.
FYI
MRUnit is the way to go and it is compatible (as far as I am aware of)
with the 0.20.2
Cheers
Olivier
Varene Olivier a écrit :
> Thanks a lot Aaron,
>
> I was looking into it, I just hope, it will be compliant with 0.20.2 ...
>
> Otherwise, I am stuck :(
>
> Cheers
> Olivier
>
> Aaron Kimball a écrit :
>> Varene,
>>
>> You might want to check out MRUnit. It's a unit test harness that
>> contains
>> mock objects for the context & other associated classes, and works with
>> JUnit.
>>
>> It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
>> Distribution for Hadoop. See
>> http://archive.cloudera.com/docs/mrunit/index.html for MRUnit
>> documentation.
>>
>> Cheers,
>> - Aaron
>>
>> On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <va...@echo.fr> wrote:
>>
>>> FYI
>>> I am on Hadoop 0.20.2, JUnit 4
>>>
>>>
>>> Varene Olivier a écrit :
>>>
>>> Hello to all,
>>>> first of all many thanks for this great piece of software you are all
>>>> contributing to :)
>>>>
>>>> I am actually creating a program in Hadoop MapReduce, but I intend to
>>>> massively use JUnit Tests to validate my code.
>>>>
>>>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>>>> function with it, but inside my Mapper.map function, I am using
>>>> Counters
>>>> to maintain internal statistics. The update of those counters make
>>>> the test
>>>> fails with a null pointer (JavaNullPointerException).
>>>> There is an issue "Counters" of variable visibility from my test ...
>>>>
>>>> Do you have any idea how to mock/validate/overcome this issue ?
>>>>
>>>>
>>>>
>>>> Thanks a lots
>>>> Best Regards
>>>> Olivier Varene
>>>>
>>>>
>>>> : Code :
>>>> --------
>>>>
>>>> private static enum Counters {NBLINES};
>>>> private static IntWritable one = new IntWritable(1);
>>>> private static LongWritable oneL = new LongWritable(1);
>>>>
>>>>
>>>> // Mapper
>>>> /**
>>>> * outputs 1 for each line encountered
>>>> */
>>>> public static class LCMapper extends
>>>> Mapper<LongWritable, Text, IntWritable, LongWritable>
>>>> {
>>>> private final static IntWritable one = new IntWritable(1);
>>>> private final static LongWritable oneL = new LongWritable(1);
>>>> /**
>>>> * for each line encountered outputs 1 for the key 1
>>>> */
>>>> public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>>> throws IOException, InterruptedException
>>>> {
>>>> p_context.getCounter(Counters.NBLINES).increment(1);
>>>> p_context.write(one,oneL); } // end map
>>>> } // end Mapper
>>>>
>>>> : Test :
>>>> --------
>>>>
>>>> @Test
>>>> public void testLCMapper() throws IOException, InterruptedException
>>>> { LCMapper mapper = new LCMapper();
>>>> Text value = new Text("ligneA\nlignB\n");
>>>> // mock the map job execution context
>>>> Context context = mock(Context.class);
>>>> try {
>>>> mapper.map(null, value, context);
>>>> } catch (NullPointerException e)
>>>> {
>>>> // exception raised by context.getCounter(...) null pointer
>>>> // HOW TO MOCK ???
>>>> // Counter is a static enum from the englobing class ...
>>>> // issue with variable visibility (from map function) in this test
>>>> }
>>>> // verify that the Mapper issued (one,oneL) pair
>>>> verify(context).write(one,oneL);
>>>> } // end testLCMapper
>>>>
>>>>
>>>>
>>
Re: Mapper Reducer : Unit Test and mocking with static variables
Posted by Varene Olivier <va...@echo.fr>.
Thanks a lot Aaron,
I was looking into it, I just hope, it will be compliant with 0.20.2 ...
Otherwise, I am stuck :(
Cheers
Olivier
Aaron Kimball a écrit :
> Varene,
>
> You might want to check out MRUnit. It's a unit test harness that contains
> mock objects for the context & other associated classes, and works with
> JUnit.
>
> It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
> Distribution for Hadoop. See
> http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.
>
> Cheers,
> - Aaron
>
> On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <va...@echo.fr> wrote:
>
>> FYI
>> I am on Hadoop 0.20.2, JUnit 4
>>
>>
>> Varene Olivier a écrit :
>>
>> Hello to all,
>>> first of all many thanks for this great piece of software you are all
>>> contributing to :)
>>>
>>> I am actually creating a program in Hadoop MapReduce, but I intend to
>>> massively use JUnit Tests to validate my code.
>>>
>>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>>> function with it, but inside my Mapper.map function, I am using Counters
>>> to maintain internal statistics. The update of those counters make the test
>>> fails with a null pointer (JavaNullPointerException).
>>> There is an issue "Counters" of variable visibility from my test ...
>>>
>>> Do you have any idea how to mock/validate/overcome this issue ?
>>>
>>>
>>>
>>> Thanks a lots
>>> Best Regards
>>> Olivier Varene
>>>
>>>
>>> : Code :
>>> --------
>>>
>>> private static enum Counters {NBLINES};
>>> private static IntWritable one = new IntWritable(1);
>>> private static LongWritable oneL = new LongWritable(1);
>>>
>>>
>>> // Mapper
>>> /**
>>> * outputs 1 for each line encountered
>>> */
>>> public static class LCMapper extends
>>> Mapper<LongWritable, Text, IntWritable, LongWritable>
>>> {
>>> private final static IntWritable one = new IntWritable(1);
>>> private final static LongWritable oneL = new LongWritable(1);
>>> /**
>>> * for each line encountered outputs 1 for the key 1
>>> */
>>> public void map(LongWritable p_key, Text p_inputs, Context p_context)
>>> throws IOException, InterruptedException
>>> {
>>> p_context.getCounter(Counters.NBLINES).increment(1);
>>> p_context.write(one,oneL); } // end map
>>> } // end Mapper
>>>
>>> : Test :
>>> --------
>>>
>>> @Test
>>> public void testLCMapper() throws IOException, InterruptedException
>>> { LCMapper mapper = new LCMapper();
>>> Text value = new Text("ligneA\nlignB\n");
>>> // mock the map job execution context
>>> Context context = mock(Context.class);
>>> try {
>>> mapper.map(null, value, context);
>>> } catch (NullPointerException e)
>>> {
>>> // exception raised by context.getCounter(...) null pointer
>>> // HOW TO MOCK ???
>>> // Counter is a static enum from the englobing class ...
>>> // issue with variable visibility (from map function) in this test
>>> }
>>> // verify that the Mapper issued (one,oneL) pair
>>> verify(context).write(one,oneL);
>>> } // end testLCMapper
>>>
>>>
>>>
>
Re: Mapper Reducer : Unit Test and mocking with static variables
Posted by Aaron Kimball <aa...@cloudera.com>.
Varene,
You might want to check out MRUnit. It's a unit test harness that contains
mock objects for the context & other associated classes, and works with
JUnit.
It's included in the (unreleased) Hadoop 0.21, as well as Cloudera's
Distribution for Hadoop. See
http://archive.cloudera.com/docs/mrunit/index.html for MRUnit documentation.
Cheers,
- Aaron
On Thu, May 27, 2010 at 9:13 AM, Varene Olivier <va...@echo.fr> wrote:
> FYI
> I am on Hadoop 0.20.2, JUnit 4
>
>
> Varene Olivier a écrit :
>
> Hello to all,
>>
>> first of all many thanks for this great piece of software you are all
>> contributing to :)
>>
>> I am actually creating a program in Hadoop MapReduce, but I intend to
>> massively use JUnit Tests to validate my code.
>>
>> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
>> function with it, but inside my Mapper.map function, I am using Counters
>> to maintain internal statistics. The update of those counters make the test
>> fails with a null pointer (JavaNullPointerException).
>> There is an issue "Counters" of variable visibility from my test ...
>>
>> Do you have any idea how to mock/validate/overcome this issue ?
>>
>>
>>
>> Thanks a lots
>> Best Regards
>> Olivier Varene
>>
>>
>> : Code :
>> --------
>>
>> private static enum Counters {NBLINES};
>> private static IntWritable one = new IntWritable(1);
>> private static LongWritable oneL = new LongWritable(1);
>>
>>
>> // Mapper
>> /**
>> * outputs 1 for each line encountered
>> */
>> public static class LCMapper extends
>> Mapper<LongWritable, Text, IntWritable, LongWritable>
>> {
>> private final static IntWritable one = new IntWritable(1);
>> private final static LongWritable oneL = new LongWritable(1);
>> /**
>> * for each line encountered outputs 1 for the key 1
>> */
>> public void map(LongWritable p_key, Text p_inputs, Context p_context)
>> throws IOException, InterruptedException
>> {
>> p_context.getCounter(Counters.NBLINES).increment(1);
>> p_context.write(one,oneL); } // end map
>> } // end Mapper
>>
>> : Test :
>> --------
>>
>> @Test
>> public void testLCMapper() throws IOException, InterruptedException
>> { LCMapper mapper = new LCMapper();
>> Text value = new Text("ligneA\nlignB\n");
>> // mock the map job execution context
>> Context context = mock(Context.class);
>> try {
>> mapper.map(null, value, context);
>> } catch (NullPointerException e)
>> {
>> // exception raised by context.getCounter(...) null pointer
>> // HOW TO MOCK ???
>> // Counter is a static enum from the englobing class ...
>> // issue with variable visibility (from map function) in this test
>> }
>> // verify that the Mapper issued (one,oneL) pair
>> verify(context).write(one,oneL);
>> } // end testLCMapper
>>
>>
>>
Re: Mapper Reducer : Unit Test and mocking with static variables
Posted by Varene Olivier <va...@echo.fr>.
FYI
I am on Hadoop 0.20.2, JUnit 4
Varene Olivier a écrit :
> Hello to all,
>
> first of all many thanks for this great piece of software you are all
> contributing to :)
>
> I am actually creating a program in Hadoop MapReduce, but I intend to
> massively use JUnit Tests to validate my code.
>
> To test a Mapper, I mock a Mapper.Context object and run the Mapper.map
> function with it, but inside my Mapper.map function, I am using Counters
> to maintain internal statistics. The update of those counters make the
> test fails with a null pointer (JavaNullPointerException).
> There is an issue "Counters" of variable visibility from my test ...
>
> Do you have any idea how to mock/validate/overcome this issue ?
>
>
>
> Thanks a lots
> Best Regards
> Olivier Varene
>
>
> : Code :
> --------
>
> private static enum Counters {NBLINES};
> private static IntWritable one = new IntWritable(1);
> private static LongWritable oneL = new LongWritable(1);
>
>
> // Mapper
> /**
> * outputs 1 for each line encountered
> */
> public static class LCMapper extends
> Mapper<LongWritable, Text, IntWritable, LongWritable>
> {
> private final static IntWritable one = new IntWritable(1);
> private final static LongWritable oneL = new LongWritable(1);
> /**
> * for each line encountered outputs 1 for the key 1
> */
> public void map(LongWritable p_key, Text p_inputs, Context p_context)
> throws IOException, InterruptedException
> {
> p_context.getCounter(Counters.NBLINES).increment(1);
> p_context.write(one,oneL);
> } // end map
> } // end Mapper
>
> : Test :
> --------
>
> @Test
> public void testLCMapper() throws IOException, InterruptedException
> {
> LCMapper mapper = new LCMapper();
> Text value = new Text("ligneA\nlignB\n");
> // mock the map job execution context
> Context context = mock(Context.class);
> try {
> mapper.map(null, value, context);
> } catch (NullPointerException e)
> {
> // exception raised by context.getCounter(...) null pointer
> // HOW TO MOCK ???
> // Counter is a static enum from the englobing class ...
> // issue with variable visibility (from map function) in this test
> }
> // verify that the Mapper issued (one,oneL) pair
> verify(context).write(one,oneL);
>
> } // end testLCMapper
>
>