sharing data structure among mappers or reducers

[Denim Live <de...@yahoo.com>]

Is it possible to share a datastructures such as linked list or array among the 
mapper (or reducer) nodes. I am not asking about sharing files, which can be 
done by distributed cache. I want to share some data structure.


      

RE: sharing data structure among mappers or reducers

[Sanjay Sharma <sa...@impetus.co.in>]

Maps and reducers run in separate JVMs so you will have to think of a distributed way to share data (data structures or files).
The distributed options can be a distributed cache solution such as memcached or Hbase or Tokyotyrant/BDB or a simple file on HDFS. You can serialize your java objects including data structures in the cache as values.
With any cache solution, you would have to incur the cost of ser-de as well as distribution. Updating value objects in the cache is possible- it will have costs/overhead of its own, but remember that map reduce is all about stateless programming
In short, data structure sharing might not be a good idea at all- until unless it is a read-only cache, else rethink the algorithm.
- Sanjay


-----Original Message-----
From: Denim Live [mailto:denim.live@yahoo.com]
Sent: Monday, July 26, 2010 6:49 PM
To: common-user@hadoop.apache.org
Subject: sharing data structure among mappers or reducers

Is it possible to share a datastructures such as linked list or array among the
mapper (or reducer) nodes. I am not asking about sharing files, which can be
done by distributed cache. I want to share some data structure.




Meet Impetus at the OSCON 2010 in Portland, Oregon during July 19th to 23rd. Listen to our Senior Director of Engineering and expert speaker Vineet Tyagi talk about building a Ruby application server.

Click http://www.impetus.com/ to know more. Follow our updates on www.twitter.com/impetuscalling .


NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.