You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by 谭军 <ta...@163.com> on 2011/08/15 11:14:11 UTC
Can I define a datastructure that all Mappers share?
Hi,
I want to define a matrix or list that all mappers share.
So that all mappers can do operations on it.
Can I make it?
Thanks!
--
Regards!
Jun Tan
Re:Re: Can I define a datastructure that all Mappers share?
Posted by 谭军 <ta...@163.com>.
Hi,
I want to construct a cross-linklist to store a graph
I want to get (node, neighbor-list) from (node, node) records in database file.
Each Mapper processes one node.
And each Mapper gets (node, node) records and passes (node, neighbor-list) records to Reducer.
How can I make it?
--
Regards!
Jun Tan
At 2011-08-15 18:25:06,"Sudharsan Sampath" <su...@gmail.com> wrote:
Hi,
To my knowledge, its not possible with plain map-reduce. But you can try using a distributed cache on top of it. To quote a few try, hazelcast (if ur prog lang is java) or gigaspace.
Just a note, why would you want to share date across mappers. It defeats the basic assumption of map-reduce that mappers are independent work chunks.
-Sudhan S
2011/8/15 谭军<ta...@163.com>
Hi,
I want to define a matrix or list that all mappers share.
So that all mappers can do operations on it.
Can I make it?
Thanks!
--
Regards!
Jun Tan
Re: Can I define a datastructure that all Mappers share?
Posted by Sudharsan Sampath <su...@gmail.com>.
Hi,
To my knowledge, its not possible with plain map-reduce. But you can try
using a distributed cache on top of it. To quote a few try, hazelcast (if ur
prog lang is java) or gigaspace.
Just a note, why would you want to share date across mappers. It defeats the
basic assumption of map-reduce that mappers are independent work chunks.
-Sudhan S
2011/8/15 谭军 <ta...@163.com>
> Hi,
> I want to define a matrix or list that all mappers share.
> So that all mappers can do operations on it.
> Can I make it?
> Thanks!
>
>
> --
>
> Regards!
>
> Jun Tan
>
>
>