Broadcasting some data to all nodes

[Hamid Oliaei <ol...@gmail.com>]

Hi,

I want to send some data and messages to all nodes after  I run a MR job.
then begin another job.
Is there any straight way to broadcast data under hadoop framework?

Re: Broadcasting some data to all nodes

[Hamid Oliaei <ol...@gmail.com>]

Hi,

I tested DistributedCache module for copying some data to all nodes.
Unfortunately, it was time-consuming and runtime is important for my work.
I want to write a map-reduce job so that a copy of input data are generated
in output of all reducers.
Is that possible? How?
I mean i want to have copies of some data to the number of reducers.

Thanks,
Hamid

Re: Broadcasting some data to all nodes

[Hamid Oliaei <ol...@gmail.com>]

Thanks a lot. I am looking at it.

Regards,

Oliaei.

On Mon, Jan 30, 2012 at 3:47 PM, Ioan Eugen Stan <st...@gmail.com>wrote:

> Pe 30.01.2012 14:10, Hamid Oliaei a scris:
>
>  Hi,
>>
>> I want to send some data and messages to all nodes after  I run a MR job.
>> then begin another job.
>> Is there any straight way to broadcast data under hadoop framework?
>>
>>
> See if DistributedCahche [1] can do what you need. It makes application
> specific files easy to distribute.
>
> http://hadoop.apache.org/**common/docs/r0.20.2/mapred_**
> tutorial.html#DistributedCache<http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#DistributedCache>
>
> --
> Ioan Eugen Stan
> http://ieugen.blogspot.com
>

Re: Broadcasting some data to all nodes

[Ioan Eugen Stan <st...@gmail.com>]

Pe 30.01.2012 14:10, Hamid Oliaei a scris:
> Hi,
>
> I want to send some data and messages to all nodes after  I run a MR job.
> then begin another job.
> Is there any straight way to broadcast data under hadoop framework?
>

See if DistributedCahche [1] can do what you need. It makes application 
specific files easy to distribute.

http://hadoop.apache.org/common/docs/r0.20.2/mapred_tutorial.html#DistributedCache

-- 
Ioan Eugen Stan
http://ieugen.blogspot.com