how to pass values from mapper/reducer to driver code?

[Sujit Pal <su...@comcast.net>]

Hi,

I am using Hadoop 0.20. I can pass input values from the driver code
into the mapper/reducer code using the Configuration, something like
this:

// driver code:
Job job = new Job(conf, "name");
job.getConfiguration("foo", "bar");

// and in the mapper/reducer class, retrieve it like so:
String foo = context.getConfiguration().get("foo");

I am calculating some values from within my mapper as a side effect.
Regardless of which mapper drops it, it is going to be the same (ie it
is not affected by the split size). So I tried doing something like
this:

In the mapper's cleanup() method, I have:

context.getConfiguration().set("r", "result");

but when I try to retrieve it on the driver:
String result = job.getConfiguration().get("r");

I get a null in the result, so it appears as though I am either using
the wrong data structure and that 2-way communication is not supported.

Are there any other ways of getting at this sort of data? I read about
the Reporter, and I thought that perhaps I could use that, but not sure
how to do that.

Any pointers would be greatly appreciated.

TIA
Sujit

Re: how to pass values from mapper/reducer to driver code?

[Sujit Pal <su...@comcast.net>]

Thanks Anthony, that was what I was afraid it would be...

-sujit

On Wed, 2009-09-23 at 20:37 -0700, Anthony Urso wrote:
> Changing job configuration on a map process will only be visible on
> that process.
> 
> The expected communication vector between Mappers and Reducers is
> OutputCollector.collect().
> 
> Cheers,
> Anthony
> 
> On Wed, Sep 23, 2009 at 5:49 PM, Sujit Pal <su...@comcast.net> wrote:
> > Hi,
> >
> > I am using Hadoop 0.20. I can pass input values from the driver code
> > into the mapper/reducer code using the Configuration, something like
> > this:
> >
> > // driver code:
> > Job job = new Job(conf, "name");
> > job.getConfiguration("foo", "bar");
> >
> > // and in the mapper/reducer class, retrieve it like so:
> > String foo = context.getConfiguration().get("foo");
> >
> > I am calculating some values from within my mapper as a side effect.
> > Regardless of which mapper drops it, it is going to be the same (ie it
> > is not affected by the split size). So I tried doing something like
> > this:
> >
> > In the mapper's cleanup() method, I have:
> >
> > context.getConfiguration().set("r", "result");
> >
> > but when I try to retrieve it on the driver:
> > String result = job.getConfiguration().get("r");
> >
> > I get a null in the result, so it appears as though I am either using
> > the wrong data structure and that 2-way communication is not supported.
> >
> > Are there any other ways of getting at this sort of data? I read about
> > the Reporter, and I thought that perhaps I could use that, but not sure
> > how to do that.
> >
> > Any pointers would be greatly appreciated.
> >
> > TIA
> > Sujit
> >
> >
> >

Re: how to pass values from mapper/reducer to driver code?

[Anthony Urso <an...@gmail.com>]

Changing job configuration on a map process will only be visible on
that process.

The expected communication vector between Mappers and Reducers is
OutputCollector.collect().

Cheers,
Anthony

On Wed, Sep 23, 2009 at 5:49 PM, Sujit Pal <su...@comcast.net> wrote:
> Hi,
>
> I am using Hadoop 0.20. I can pass input values from the driver code
> into the mapper/reducer code using the Configuration, something like
> this:
>
> // driver code:
> Job job = new Job(conf, "name");
> job.getConfiguration("foo", "bar");
>
> // and in the mapper/reducer class, retrieve it like so:
> String foo = context.getConfiguration().get("foo");
>
> I am calculating some values from within my mapper as a side effect.
> Regardless of which mapper drops it, it is going to be the same (ie it
> is not affected by the split size). So I tried doing something like
> this:
>
> In the mapper's cleanup() method, I have:
>
> context.getConfiguration().set("r", "result");
>
> but when I try to retrieve it on the driver:
> String result = job.getConfiguration().get("r");
>
> I get a null in the result, so it appears as though I am either using
> the wrong data structure and that 2-way communication is not supported.
>
> Are there any other ways of getting at this sort of data? I read about
> the Reporter, and I thought that perhaps I could use that, but not sure
> how to do that.
>
> Any pointers would be greatly appreciated.
>
> TIA
> Sujit
>
>
>