mrunit question

[David Rosenstrauch <da...@darose.net>]

I'm trying to write some tests with the mrunit framework, but running 
into a snag.

It seems that the mock Context objects that are being created are always 
using a new, empty Configuration object.  However, I've written my 
reducer to implement Configurable, since in production I'm going to be 
passing certain specific parameters in to it as properties set on the 
Configuration object.  But with the Context always passing it an empty 
Configuration, the Reducer will fail the tests.

What I'm looking for is a way to initialize the reducer using a 
particular Configuration object, which contains the parms/properties 
that the reducer needs.  I looked at the ReduceDriver class, however, 
and there doesn't seem to be any way to do that.

I suppose I could explicitly call my reducer's setConf() method before I 
run my test, in order to give it a correct configuration.  However, it 
looks like what'll happen is that then later setConf() will get called a 
second time with the empty configuration, which will cause an exception 
getting thrown by my reducer.  (Unless I mod my code to avoid this.)

Is there any way around this conundrum without my being forced to add a 
hack to my reducer for this?  I.e., any way to initialize the mrunit 
framework to use a specific config when it runs a test?

TIA,

DR

Re: mrunit question

[David Rosenstrauch <da...@darose.net>]

On 08/11/2010 08:14 PM, Aaron Kimball wrote:
> David,
>
> Since you are directly instantiating the Mapper and Reducer (not using
> ReflectionUtils), you are free to call setConf() yourself before you run the
> test.

Sort of.  What would wind up happening is that setConf would get called 
twice:  once by the mrunit framework with the empty config (for which 
I'd need to add special coding in to my reducer object to make it ignore 
it) and then a second time when I explicitly call it myself in my JUnit 
test.  It's not really ideal for me to have to add special code into my 
reducer to avoid this issue.


> If you're using the new API, there already is setConfiguration() and
> withConfiguration(), which will set the Configuration being passed into
> Context sent to your Mapper/Reducer's run() method.
> See https://issues.apache.org/jira/browse/mapreduce-1569 where this was
> introduced.
>
>
> - Aaron

Yes, just looked at that, and it looks like it exactly solves this 
issue.  Unfortunately, though, it says that fix is effective in v0.21. 
As we're running the stable CDH version (CDH2, which uses Hadoop v0.20) 
I can't take advantage of that fix yet.

Going to wind up forgoing unit tests on this M/R code for right now I think.

Thanks,

DR

Re: mrunit question

[Aaron Kimball <aa...@cloudera.com>]

David,

Since you are directly instantiating the Mapper and Reducer (not using
ReflectionUtils), you are free to call setConf() yourself before you run the
test.

If you're using the old API (o.a.h.mrunit):

Mapper m = new Mapper();
MapDriver d = new MapDriver(m);
Configuration conf = new Configuration();
conf.set("foo", "bar");
m.setConf(conf);
d.withInput(k, v).withOutput(k2, v2).runTest();


If you're using the new API, there already is setConfiguration() and
withConfiguration(), which will set the Configuration being passed into
Context sent to your Mapper/Reducer's run() method.
See https://issues.apache.org/jira/browse/mapreduce-1569 where this was
introduced.


- Aaron

On Tue, Aug 10, 2010 at 7:44 PM, David Rosenstrauch <da...@darose.net>wrote:

> On 08/10/2010 04:54 PM, David Rosenstrauch wrote:
>
>> On 08/10/2010 04:24 PM, David Rosenstrauch wrote:
>>
>>> What I'm looking for is a way to initialize the reducer using a
>>> particular Configuration object, which contains the parms/properties
>>> that the reducer needs. I looked at the ReduceDriver class, however, and
>>> there doesn't seem to be any way to do that.
>>>
>>
>> Obvious way to solve thi issue would be an enhancement to ReduceDriver
>> to add setConfiguration/withConfiguration methods. (Along with a similar
>> enhancement to the other drivers.) This sound useful to anyone else?
>>
>> Thanks,
>>
>> DR
>>
>
> FYI - I've filed a bug report on this:
>
> https://issues.apache.org/jira/browse/MAPREDUCE-2002
>
> Thanks,
>
> DR
>

Re: mrunit question

[David Rosenstrauch <da...@darose.net>]

On 08/10/2010 04:54 PM, David Rosenstrauch wrote:
> On 08/10/2010 04:24 PM, David Rosenstrauch wrote:
>> What I'm looking for is a way to initialize the reducer using a
>> particular Configuration object, which contains the parms/properties
>> that the reducer needs. I looked at the ReduceDriver class, however, and
>> there doesn't seem to be any way to do that.
>
> Obvious way to solve thi issue would be an enhancement to ReduceDriver
> to add setConfiguration/withConfiguration methods. (Along with a similar
> enhancement to the other drivers.) This sound useful to anyone else?
>
> Thanks,
>
> DR

FYI - I've filed a bug report on this:

https://issues.apache.org/jira/browse/MAPREDUCE-2002

Thanks,

DR

Re: mrunit question

[David Rosenstrauch <da...@darose.net>]

On 08/10/2010 04:24 PM, David Rosenstrauch wrote:
> What I'm looking for is a way to initialize the reducer using a
> particular Configuration object, which contains the parms/properties
> that the reducer needs. I looked at the ReduceDriver class, however, and
> there doesn't seem to be any way to do that.

Obvious way to solve thi issue would be an enhancement to ReduceDriver 
to add setConfiguration/withConfiguration methods.  (Along with a 
similar enhancement to the other drivers.)  This sound useful to anyone 
else?

Thanks,

DR