static data sharing among map funtions

[Wasim Bari <wa...@msn.com>]

 

 

Hi,

   How static data is shared among mappers ? is it synchorinized or not possible ? I will appreciate if someone can give possible way of sharing data among Map funtions in one job.

 

BR,

 

Bari

Re: static data sharing among map funtions

[brien colwell <xc...@gmail.com>]

Here's a setup I've used:
- configuration data distributed to the mappers / reducers using the 
JobConf object
- BDBs (stored in ZIP packages on the HDFS) used for read/write data 
across stages. The data flow organized so a single mapper modifies a 
single database per stage, to avoid concurrency issues.

The concurrency of the shared read/write data will affect the storage 
type. In general it's better performance to as much data local as 
possible and then distribute it (e.g. store on the HDFS) at the end of 
the mapper job. If you need all mappers to share the same data at once, 
then a technology like memcache seems like a good approach.





Chandraprakash Bhagtani wrote:
> If you really want to share read/write data you can use memcached server or
> file based database like Tokyocabinet or BDB
>
>
>   

Re: static data sharing among map funtions

[Chandraprakash Bhagtani <cp...@gmail.com>]

If you really want to share read/write data you can use memcached server or
file based database like Tokyocabinet or BDB


-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

On Thu, Sep 10, 2009 at 2:22 PM, indoos <in...@gmail.com> wrote:

>
> Hi,
> Here are 2 possible ways for static data sharing-
> 1. Using distributed cache- refer
>
> http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#DistributedCache
> 2. Using JobConf object-
>
> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/conf/Configuration.html#set%28java.lang.String,%20java.lang.String%29
> .
>
> Please note that only read only cache should be used in Hadoop.
>
> Regards,
> Sanjay
>
>
> wbari wrote:
> >
> >
> >
> >
> >
> >
> > Hi,
> >
> >    How static data is shared among mappers ? is it synchorinized or not
> > possible ? I will appreciate if someone can give possible way of sharing
> > data among Map funtions in one job.
> >
> >
> >
> > BR,
> >
> >
> >
> > Bari
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/static-data-sharing-among-map-funtions-tp25347960p25379620.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

Re: static data sharing among map funtions

[indoos <in...@gmail.com>]

Hi,
Here are 2 possible ways for static data sharing-
1. Using distributed cache- refer
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#DistributedCache
2. Using JobConf object-
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/conf/Configuration.html#set%28java.lang.String,%20java.lang.String%29.   

Please note that only read only cache should be used in Hadoop.

Regards,
Sanjay


wbari wrote:
> 
> 
>  
> 
>  
> 
> Hi,
> 
>    How static data is shared among mappers ? is it synchorinized or not
> possible ? I will appreciate if someone can give possible way of sharing
> data among Map funtions in one job.
> 
>  
> 
> BR,
> 
>  
> 
> Bari
> 
> 

-- 
View this message in context: http://www.nabble.com/static-data-sharing-among-map-funtions-tp25347960p25379620.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.