Re: Passing messages

[Arun C Murthy <ac...@yahoo-inc.com>]

Counters are really not designed for what you want...

I'll warn you that you will need to construct a new JobClient to fetch  
the Counters. They just won't show up if you get counters from the  
Reporter in the reduce - they are local ones only. Nor can you get per- 
taskattempt Counters from the JT - you only get globally aggregated  
ones.

So, ZK is probably your only choice.

Arun

On Dec 20, 2010, at 12:57 AM, Martin Becker wrote:

> Thank you for your suggestions. In this context I heard about
> ZooKeeper a few times. It seems to be the easiest and most failsafe
> solution as of yet. Another solution mentioned was using some sort of
> communication through the file system, which is probably slow and is
> quite subtle. Of course, I would like to see a Hadoop/MapReduce
> internal solution. I will give the counters a try, first. If that does
> not work I will use ZooKeeper instead. All the while hoping for any
> Hadoop/MapReduce internal solution to come up.
>
> On Mon, Dec 20, 2010 at 4:24 AM, David Rosenstrauch  
> <da...@darose.net> wrote:
>> On 12/18/2010 11:04 AM, Martin Becker wrote:
>>>
>>> Hello everbody,
>>>
>>> I am wondering if there is a feature allowing (in my case) reduce
>>> tasks to communicate. For example by some volatile variables at some
>>> centralized point. Or maybe just notify other running or to-be- 
>>> running
>>> reduce tasks of a completed reduce task featuring some arguments.
>>>
>>> In my case, I have reduce tasks doing computations that will
>>> output/produce certain quality threshold. Other reduce tasks can/ 
>>> could
>>> estimate, if they ever get above those thresholds. If not they could
>>> just cease running.
>>>
>>> Thanks in advance,
>>> Martin
>>>
>>> PS: If such functions are not yet part of the API, I would like to
>>> know if there are good reasons for it and if not, propose to  
>>> introduce
>>> such functionality.
>>
>> You could use Zookeeper to do this.
>>
>> DR
>>