You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mrunit.apache.org by jamal sasha <ja...@gmail.com> on 2013/10/27 01:13:54 UTC
passing configuration in mrunit
Hi,
I have been searching in mrunit documentation but hasnt been able to find
it so far..
How do i pass configuration parameters in my mrunit.
Hi,
I have been searching in mrunit documentation but hasnt been able to find
it so far..
How do i pass configuration parameters in my mrunit.
So for example, if i take the wordcount example.
Lets say, in my driver code I am setting this parameter...
conf.set("delimiter",args[2])
And in my mapper code I am calling this as:
String delimiter = conf.get("delimiter");
String [] tokens = value.toString().split(delimiter);
for (String token:tokens)
context.write(token,one);
How do I set up this configuration parameter.
I have been looking into this example:
https://github.com/wpm/Hadoop-Word-Count/blob/master/src/test/java/wpmcn/hadoop/WordCountTest.java
Thanks
Re: passing configuration in mrunit
Posted by Tom Wheeler <tw...@cloudera.com>.
On Sun, Oct 27, 2013 at 12:12 AM, jamal sasha <ja...@gmail.com> wrote:
> Eh another noob question.
> In my mapper... that "context".. object is not recognized..?
> How do i parse that?
> Thanks for all this help :)
It ought to be recognized, if you're using the Mapper class from here:
https://github.com/wpm/Hadoop-Word-Count/blob/master/src/main/java/wpmcn/hadoop/WordCount.java
As you can see, the Context object is passed as a parameter in the map() method.
If what you meant to ask is "how can I get the Configuration object
from the method that tests my Mapper (instead of from the map method
itself)?" then you should be able to do that like this:
Configuration conf = mapDriver.getConfiguration();
Re: passing configuration in mrunit
Posted by jamal sasha <ja...@gmail.com>.
Eh another noob question.
In my mapper... that "context".. object is not recognized..?
How do i parse that?
Thanks for all this help :)
On Sat, Oct 26, 2013 at 7:34 PM, Tom Wheeler <tw...@cloudera.com> wrote:
> Hi Jamal,
>
> Let's keep it simple at first and concentrate just on getting the
> configuration to your mapper. In the setUp() method you posted for
> your unit test, you already have these lines:
>
> Configuration conf = new Configuration();
> conf.set("delimiter",",");
>
> What I was saying before is that you need to add just one more line after
> those:
>
> mapDriver.setConfiguration(conf);
>
> Doing that will make the configuration options you set there available
> to your mapper. You can check that it worked by putting code like this
> in your mapper's map method:
>
> Configuration conf = context.getConfiguration();
> // you should see that this shows a comma when you run your test
> System.out.println("Here is the delimiter I set: " +
> conf.get("delimiter"));
>
> Once you get that working, you can set the configuration for your
> reduce driver and map-reduce driver by adding the following to your
> setUp() method:
>
> reduceDriver.setConfiguration(conf);
> mapReduceDriver.setConfiguration(conf);
>
> Hope that helps,
>
> Tom
>
> On Sat, Oct 26, 2013 at 7:20 PM, jamal sasha <ja...@gmail.com>
> wrote:
> > Hi
> > Thanks for replying...
> > I am not sure if I have got that.. So, my apologies for asking noob
> > questions.
> >
> > How do I get that conf object?
> > So for example:
> >
> > @Before
> > public void setUp() {
> > WordCount.WordCountMapper mapper = new WordCount.WordCountMapper();
> > WordCount.WordCountReducer reducer = new
> WordCount.WordCountReducer();
> > mapDriver = new MapDriver<LongWritable, Text, Text,
> LongWritable>();
> > mapDriver.setMapper(mapper);
> > reduceDriver = new ReduceDriver<Text, LongWritable, Text,
> > LongWritable>();
> > reduceDriver.setReducer(reducer);
> > mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text,
> > LongWritable, Text, LongWritable>();
> > mapReduceDriver.setMapper(mapper);
> > mapReduceDriver.setReducer(reducer);
> > }
> >
> >
> >
> > Now one of the configuration params I want to pass to mapper is
> "delimiter"
> > My mapreduce code has a setup method where I am getting this parameter...
> > But I am not sure how to pass this in mrunit..
> > So in my setup.. Does this makes sense
> >
> > @Before
> > public void setUp() {
> > WordCount.WordCountMapper mapper = new WordCount.WordCountMapper();
> > WordCount.WordCountReducer reducer = new
> WordCount.WordCountReducer();
> > mapDriver = new MapDriver<LongWritable, Text, Text,
> LongWritable>();
> > Configuration conf = new Configuration();
> > conf.set("delimiter",",");
> >
> > mapDriver.setMapper(mapper);
> > reduceDriver = new ReduceDriver<Text, LongWritable, Text,
> > LongWritable>();
> > reduceDriver.setReducer(reducer);
> > mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text,
> > LongWritable, Text, LongWritable>();
> > mapReduceDriver.setMapper(mapper);
> > mapReduceDriver.setReducer(reducer);
> > }
> >
> >
> >
> >
> > Can you please fill in above stub... to set this delimiter parameter
> which
> > will be then recovered in mapper code?
> > Thanks
> >
> >
> >
> >
> >
> >
> >
> >
> > On Sat, Oct 26, 2013 at 5:10 PM, Tom Wheeler <tw...@cloudera.com>
> wrote:
> >>
> >> I think you might just be missing the line where you set the
> >> configuration on your Driver class. For example, add a line like this
> >> in the setUp() method:
> >>
> >> mapDriver.setConfiguration(conf);
> >>
> >>
> >>
> >> On Sat, Oct 26, 2013 at 6:13 PM, jamal sasha <ja...@gmail.com>
> >> wrote:
> >> > Hi,
> >> > I have been searching in mrunit documentation but hasnt been able to
> >> > find
> >> > it so far..
> >> > How do i pass configuration parameters in my mrunit.
> >> >
> >> > Hi,
> >> > I have been searching in mrunit documentation but hasnt been able to
> >> > find
> >> > it so far..
> >> > How do i pass configuration parameters in my mrunit.
> >> > So for example, if i take the wordcount example.
> >> >
> >> > Lets say, in my driver code I am setting this parameter...
> >> >
> >> > conf.set("delimiter",args[2])
> >> >
> >> > And in my mapper code I am calling this as:
> >> >
> >> > String delimiter = conf.get("delimiter");
> >> > String [] tokens = value.toString().split(delimiter);
> >> > for (String token:tokens)
> >> > context.write(token,one);
> >> >
> >> > How do I set up this configuration parameter.
> >> >
> >> > I have been looking into this example:
> >> >
> >> >
> https://github.com/wpm/Hadoop-Word-Count/blob/master/src/test/java/wpmcn/hadoop/WordCountTest.java
> >> >
> >> > Thanks
> >
> >
>
Re: passing configuration in mrunit
Posted by Tom Wheeler <tw...@cloudera.com>.
Hi Jamal,
Let's keep it simple at first and concentrate just on getting the
configuration to your mapper. In the setUp() method you posted for
your unit test, you already have these lines:
Configuration conf = new Configuration();
conf.set("delimiter",",");
What I was saying before is that you need to add just one more line after those:
mapDriver.setConfiguration(conf);
Doing that will make the configuration options you set there available
to your mapper. You can check that it worked by putting code like this
in your mapper's map method:
Configuration conf = context.getConfiguration();
// you should see that this shows a comma when you run your test
System.out.println("Here is the delimiter I set: " + conf.get("delimiter"));
Once you get that working, you can set the configuration for your
reduce driver and map-reduce driver by adding the following to your
setUp() method:
reduceDriver.setConfiguration(conf);
mapReduceDriver.setConfiguration(conf);
Hope that helps,
Tom
On Sat, Oct 26, 2013 at 7:20 PM, jamal sasha <ja...@gmail.com> wrote:
> Hi
> Thanks for replying...
> I am not sure if I have got that.. So, my apologies for asking noob
> questions.
>
> How do I get that conf object?
> So for example:
>
> @Before
> public void setUp() {
> WordCount.WordCountMapper mapper = new WordCount.WordCountMapper();
> WordCount.WordCountReducer reducer = new WordCount.WordCountReducer();
> mapDriver = new MapDriver<LongWritable, Text, Text, LongWritable>();
> mapDriver.setMapper(mapper);
> reduceDriver = new ReduceDriver<Text, LongWritable, Text,
> LongWritable>();
> reduceDriver.setReducer(reducer);
> mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text,
> LongWritable, Text, LongWritable>();
> mapReduceDriver.setMapper(mapper);
> mapReduceDriver.setReducer(reducer);
> }
>
>
>
> Now one of the configuration params I want to pass to mapper is "delimiter"
> My mapreduce code has a setup method where I am getting this parameter...
> But I am not sure how to pass this in mrunit..
> So in my setup.. Does this makes sense
>
> @Before
> public void setUp() {
> WordCount.WordCountMapper mapper = new WordCount.WordCountMapper();
> WordCount.WordCountReducer reducer = new WordCount.WordCountReducer();
> mapDriver = new MapDriver<LongWritable, Text, Text, LongWritable>();
> Configuration conf = new Configuration();
> conf.set("delimiter",",");
>
> mapDriver.setMapper(mapper);
> reduceDriver = new ReduceDriver<Text, LongWritable, Text,
> LongWritable>();
> reduceDriver.setReducer(reducer);
> mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text,
> LongWritable, Text, LongWritable>();
> mapReduceDriver.setMapper(mapper);
> mapReduceDriver.setReducer(reducer);
> }
>
>
>
>
> Can you please fill in above stub... to set this delimiter parameter which
> will be then recovered in mapper code?
> Thanks
>
>
>
>
>
>
>
>
> On Sat, Oct 26, 2013 at 5:10 PM, Tom Wheeler <tw...@cloudera.com> wrote:
>>
>> I think you might just be missing the line where you set the
>> configuration on your Driver class. For example, add a line like this
>> in the setUp() method:
>>
>> mapDriver.setConfiguration(conf);
>>
>>
>>
>> On Sat, Oct 26, 2013 at 6:13 PM, jamal sasha <ja...@gmail.com>
>> wrote:
>> > Hi,
>> > I have been searching in mrunit documentation but hasnt been able to
>> > find
>> > it so far..
>> > How do i pass configuration parameters in my mrunit.
>> >
>> > Hi,
>> > I have been searching in mrunit documentation but hasnt been able to
>> > find
>> > it so far..
>> > How do i pass configuration parameters in my mrunit.
>> > So for example, if i take the wordcount example.
>> >
>> > Lets say, in my driver code I am setting this parameter...
>> >
>> > conf.set("delimiter",args[2])
>> >
>> > And in my mapper code I am calling this as:
>> >
>> > String delimiter = conf.get("delimiter");
>> > String [] tokens = value.toString().split(delimiter);
>> > for (String token:tokens)
>> > context.write(token,one);
>> >
>> > How do I set up this configuration parameter.
>> >
>> > I have been looking into this example:
>> >
>> > https://github.com/wpm/Hadoop-Word-Count/blob/master/src/test/java/wpmcn/hadoop/WordCountTest.java
>> >
>> > Thanks
>
>
Re: passing configuration in mrunit
Posted by jamal sasha <ja...@gmail.com>.
Hi
Thanks for replying...
I am not sure if I have got that.. So, my apologies for asking noob
questions.
How do I get that conf object?
So for example:
@Before
public void setUp() {
WordCount.WordCountMapper mapper = new WordCount.WordCountMapper();
WordCount.WordCountReducer reducer = new WordCount.WordCountReducer();
mapDriver = new MapDriver<LongWritable, Text, Text, LongWritable>();
mapDriver.setMapper(mapper);
reduceDriver = new ReduceDriver<Text, LongWritable, Text, LongWritable>();
reduceDriver.setReducer(reducer);
mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text,
LongWritable, Text, LongWritable>();
mapReduceDriver.setMapper(mapper);
mapReduceDriver.setReducer(reducer);
}
Now one of the configuration params I want to pass to mapper is "delimiter"
My mapreduce code has a setup method where I am getting this parameter...
But I am not sure how to pass this in mrunit..
So in my setup.. Does this makes sense
@Before
public void setUp() {
WordCount.WordCountMapper mapper = new WordCount.WordCountMapper();
WordCount.WordCountReducer reducer = new WordCount.WordCountReducer();
mapDriver = new MapDriver<LongWritable, Text, Text, LongWritable>();
Configuration conf = new Configuration();
conf.set("delimiter",",");
mapDriver.setMapper(mapper);
reduceDriver = new ReduceDriver<Text, LongWritable, Text, LongWritable>();
reduceDriver.setReducer(reducer);
mapReduceDriver = new MapReduceDriver<LongWritable, Text, Text,
LongWritable, Text, LongWritable>();
mapReduceDriver.setMapper(mapper);
mapReduceDriver.setReducer(reducer);
}
Can you please fill in above stub... to set this delimiter parameter which
will be then recovered in mapper code?
Thanks
On Sat, Oct 26, 2013 at 5:10 PM, Tom Wheeler <tw...@cloudera.com> wrote:
> I think you might just be missing the line where you set the
> configuration on your Driver class. For example, add a line like this
> in the setUp() method:
>
> mapDriver.setConfiguration(conf);
>
>
>
> On Sat, Oct 26, 2013 at 6:13 PM, jamal sasha <ja...@gmail.com>
> wrote:
> > Hi,
> > I have been searching in mrunit documentation but hasnt been able to
> find
> > it so far..
> > How do i pass configuration parameters in my mrunit.
> >
> > Hi,
> > I have been searching in mrunit documentation but hasnt been able to
> find
> > it so far..
> > How do i pass configuration parameters in my mrunit.
> > So for example, if i take the wordcount example.
> >
> > Lets say, in my driver code I am setting this parameter...
> >
> > conf.set("delimiter",args[2])
> >
> > And in my mapper code I am calling this as:
> >
> > String delimiter = conf.get("delimiter");
> > String [] tokens = value.toString().split(delimiter);
> > for (String token:tokens)
> > context.write(token,one);
> >
> > How do I set up this configuration parameter.
> >
> > I have been looking into this example:
> >
> https://github.com/wpm/Hadoop-Word-Count/blob/master/src/test/java/wpmcn/hadoop/WordCountTest.java
> >
> > Thanks
>
Re: passing configuration in mrunit
Posted by Tom Wheeler <tw...@cloudera.com>.
I think you might just be missing the line where you set the
configuration on your Driver class. For example, add a line like this
in the setUp() method:
mapDriver.setConfiguration(conf);
On Sat, Oct 26, 2013 at 6:13 PM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> I have been searching in mrunit documentation but hasnt been able to find
> it so far..
> How do i pass configuration parameters in my mrunit.
>
> Hi,
> I have been searching in mrunit documentation but hasnt been able to find
> it so far..
> How do i pass configuration parameters in my mrunit.
> So for example, if i take the wordcount example.
>
> Lets say, in my driver code I am setting this parameter...
>
> conf.set("delimiter",args[2])
>
> And in my mapper code I am calling this as:
>
> String delimiter = conf.get("delimiter");
> String [] tokens = value.toString().split(delimiter);
> for (String token:tokens)
> context.write(token,one);
>
> How do I set up this configuration parameter.
>
> I have been looking into this example:
> https://github.com/wpm/Hadoop-Word-Count/blob/master/src/test/java/wpmcn/hadoop/WordCountTest.java
>
> Thanks