You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Geoffry Roberts <ge...@gmail.com> on 2011/05/06 19:12:55 UTC
Passing an Object to All Reducers
All,
I need for each one of my reducers to have read access to a certain object
or a clone thereof. I can instantiate this object a start up. How can I
give my reducers a copy?
--
Geoffry Roberts
Re: Passing an Object to All Reducers
Posted by David Rosenstrauch <da...@darose.net>.
On 05/06/2011 01:12 PM, Geoffry Roberts wrote:
> All,
>
> I need for each one of my reducers to have read access to a certain object
> or a clone thereof. I can instantiate this object a start up. How can I
> give my reducers a copy?
Serialize it to a string, set it as a configuration setting on the job's
configuration, then have each reducer read it from out of the config
setting, and deserialize the object.
HTH,
DR
Re: Passing an Object to All Reducers
Posted by Geoffry Roberts <ge...@gmail.com>.
Steve,
Yes the object is known at start up and is read only. The mappers don't
touch it. I considered serializing to a string. I was wondering if there
wasn't a more excellent way.
Thanks
On 6 May 2011 10:55, Steve Lewis <lo...@gmail.com> wrote:
> If possible serialize the object as XML then add it as a set of lines to
> the config - alternatively serialize it (maybe xml) to a known spot in HDFS
> and read it in in the setup code in the reducer - I assume this is an object
> known at the start of the job and not modified by the mapper
>
>
> On Fri, May 6, 2011 at 10:12 AM, Geoffry Roberts <
> geoffry.roberts@gmail.com> wrote:
>
>> All,
>>
>> I need for each one of my reducers to have read access to a certain object
>> or a clone thereof. I can instantiate this object a start up. How can I
>> give my reducers a copy?
>>
>> --
>> Geoffry Roberts
>>
>>
>
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>
--
Geoffry Roberts
Re: Passing an Object to All Reducers
Posted by Steve Lewis <lo...@gmail.com>.
If possible serialize the object as XML then add it as a set of lines to the
config - alternatively serialize it (maybe xml) to a known spot in HDFS and
read it in in the setup code in the reducer - I assume this is an object
known at the start of the job and not modified by the mapper
On Fri, May 6, 2011 at 10:12 AM, Geoffry Roberts
<ge...@gmail.com>wrote:
> All,
>
> I need for each one of my reducers to have read access to a certain object
> or a clone thereof. I can instantiate this object a start up. How can I
> give my reducers a copy?
>
> --
> Geoffry Roberts
>
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com