You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Martin Becker <_m...@web.de> on 2010/12/18 17:04:39 UTC
Passing messages
Hello everbody,
I am wondering if there is a feature allowing (in my case) reduce
tasks to communicate. For example by some volatile variables at some
centralized point. Or maybe just notify other running or to-be-running
reduce tasks of a completed reduce task featuring some arguments.
In my case, I have reduce tasks doing computations that will
output/produce certain quality threshold. Other reduce tasks can/could
estimate, if they ever get above those thresholds. If not they could
just cease running.
Thanks in advance,
Martin
PS: If such functions are not yet part of the API, I would like to
know if there are good reasons for it and if not, propose to introduce
such functionality.
Re: Passing messages
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Counters are really not designed for what you want...
I'll warn you that you will need to construct a new JobClient to fetch
the Counters. They just won't show up if you get counters from the
Reporter in the reduce - they are local ones only. Nor can you get per-
taskattempt Counters from the JT - you only get globally aggregated
ones.
So, ZK is probably your only choice.
Arun
On Dec 20, 2010, at 12:57 AM, Martin Becker wrote:
> Thank you for your suggestions. In this context I heard about
> ZooKeeper a few times. It seems to be the easiest and most failsafe
> solution as of yet. Another solution mentioned was using some sort of
> communication through the file system, which is probably slow and is
> quite subtle. Of course, I would like to see a Hadoop/MapReduce
> internal solution. I will give the counters a try, first. If that does
> not work I will use ZooKeeper instead. All the while hoping for any
> Hadoop/MapReduce internal solution to come up.
>
> On Mon, Dec 20, 2010 at 4:24 AM, David Rosenstrauch
> <da...@darose.net> wrote:
>> On 12/18/2010 11:04 AM, Martin Becker wrote:
>>>
>>> Hello everbody,
>>>
>>> I am wondering if there is a feature allowing (in my case) reduce
>>> tasks to communicate. For example by some volatile variables at some
>>> centralized point. Or maybe just notify other running or to-be-
>>> running
>>> reduce tasks of a completed reduce task featuring some arguments.
>>>
>>> In my case, I have reduce tasks doing computations that will
>>> output/produce certain quality threshold. Other reduce tasks can/
>>> could
>>> estimate, if they ever get above those thresholds. If not they could
>>> just cease running.
>>>
>>> Thanks in advance,
>>> Martin
>>>
>>> PS: If such functions are not yet part of the API, I would like to
>>> know if there are good reasons for it and if not, propose to
>>> introduce
>>> such functionality.
>>
>> You could use Zookeeper to do this.
>>
>> DR
>>
Re: Passing messages
Posted by Martin Becker <_m...@web.de>.
Thank you for your suggestions. In this context I heard about
ZooKeeper a few times. It seems to be the easiest and most failsafe
solution as of yet. Another solution mentioned was using some sort of
communication through the file system, which is probably slow and is
quite subtle. Of course, I would like to see a Hadoop/MapReduce
internal solution. I will give the counters a try, first. If that does
not work I will use ZooKeeper instead. All the while hoping for any
Hadoop/MapReduce internal solution to come up.
On Mon, Dec 20, 2010 at 4:24 AM, David Rosenstrauch <da...@darose.net> wrote:
> On 12/18/2010 11:04 AM, Martin Becker wrote:
>>
>> Hello everbody,
>>
>> I am wondering if there is a feature allowing (in my case) reduce
>> tasks to communicate. For example by some volatile variables at some
>> centralized point. Or maybe just notify other running or to-be-running
>> reduce tasks of a completed reduce task featuring some arguments.
>>
>> In my case, I have reduce tasks doing computations that will
>> output/produce certain quality threshold. Other reduce tasks can/could
>> estimate, if they ever get above those thresholds. If not they could
>> just cease running.
>>
>> Thanks in advance,
>> Martin
>>
>> PS: If such functions are not yet part of the API, I would like to
>> know if there are good reasons for it and if not, propose to introduce
>> such functionality.
>
> You could use Zookeeper to do this.
>
> DR
>
Re: Passing messages
Posted by David Rosenstrauch <da...@darose.net>.
On 12/18/2010 11:04 AM, Martin Becker wrote:
> Hello everbody,
>
> I am wondering if there is a feature allowing (in my case) reduce
> tasks to communicate. For example by some volatile variables at some
> centralized point. Or maybe just notify other running or to-be-running
> reduce tasks of a completed reduce task featuring some arguments.
>
> In my case, I have reduce tasks doing computations that will
> output/produce certain quality threshold. Other reduce tasks can/could
> estimate, if they ever get above those thresholds. If not they could
> just cease running.
>
> Thanks in advance,
> Martin
>
> PS: If such functions are not yet part of the API, I would like to
> know if there are good reasons for it and if not, propose to introduce
> such functionality.
You could use Zookeeper to do this.
DR
Re: Passing messages
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Dec 19, 2010, at 10:21 AM, Eric wrote:
> I don't know if there is such a thing in Hadoop, I'm guessing not since MapReduce is designed to have independent mappers and reducers.
Yup.
> I'm just suggesting something here: you could write a small server yourself. Say you start your jobs from a work node that has only light load. On that node, you also create something that works like memcache. (maybe you can even use memcache?)
Or just use ZooKeeper.
Re: Passing messages
Posted by Eric <er...@gmail.com>.
I don't know if there is such a thing in Hadoop, I'm guessing not since
MapReduce is designed to have independent mappers and reducers. However,
I can see the need and usefulness of some form of shared memory.
I'm just suggesting something here: you could write a small server
yourself. Say you start your jobs from a work node that has only light
load. On that node, you also create something that works like memcache.
(maybe you can even use memcache?)
From all your other nodes, you periodically check the status variables
(e.g. serialized java objects) on the memcache server. It's a single
point of failure and might not scale into the thousands but at least
it's something.
You can probably implement this with Java's RMI too.
Op 19-12-10 00:44, Martin Becker schreef:
> Well, I will have to try, if there is no other solution. I _need_
> something else, as mentioned. But if it is not supported I will have
> to stick with what is there.
>
> Yet, I also see no reason for Hadoop MapReduce, not to support some
> kind of message passing. If I miss some point here, I would like to
> know.
>
> Thanks,
> Martin
Re: Passing messages
Posted by Martin Becker <_m...@web.de>.
Well, I will have to try, if there is no other solution. I _need_
something else, as mentioned. But if it is not supported I will have
to stick with what is there.
Yet, I also see no reason for Hadoop MapReduce, not to support some
kind of message passing. If I miss some point here, I would like to
know.
Thanks,
Martin
On Sun, Dec 19, 2010 at 12:38 AM, Jason <ur...@gmail.com> wrote:
>> real time values are not required. Some lagging is tolerable. The
>> value/threshold communication is only needed to keep other reducers
>> from doing unnecessary work
>
>
> Then I think counters is what you really need here (assuming they get really updated, but I never tried that)
>
> Sent from my iPhone 4
>
> On Dec 18, 2010, at 3:19 PM, Martin Becker <_m...@web.de> wrote:
>
>> Hello Jason,
>>
>> real time values are not required. Some lagging is tolerable. The
>> value/threshold communication is only needed to keep other reducers
>> from doing unnecessary work. Some upper bound would be nice to know,
>> though. A single reducer is not an option for my algorithm. That would
>> defeat the purpose of using MapReduce. I still would like to know, if
>> there is something more general than Counters. Using other data types
>> would be convenient. Even better would be, as mentioned, some kind of
>> simple message passing system. It seems that neither is supported by
>> Hadoop as of now?
>>
>> Thank you,
>> Martin
>>
>> On Sat, Dec 18, 2010 at 10:07 PM, Jason <ur...@gmail.com> wrote:
>>>> Reducers would retrieve that increased value when accessing the same
>>>> Counter?
>>>
>>> I do not think counters reflect real time value. Even if they get updated the values will lag.
>>> If you require uptodate value I am afraid you will have to run a single reducer.
>>>
>>> Sent from my iPhone 4
>>>
>>> On Dec 18, 2010, at 10:33 AM, Martin Becker <_m...@web.de> wrote:
>>>
>>>> Thank you Ted,
>>>>
>>>> I am using the 21.0 API so I would be drawing Counters from the
>>>> Context. So if a Counter is increased on a certain Reducer other
>>>> Reducers would retrieve that increased value when accessing the same
>>>> Counter? If so, then that is an interesting piece of information.
>>>>
>>>> Unfortunately my threshold are doubles. I guess, I could find some
>>>> kind of conversion there. But is there any more general way to pass
>>>> information between Reducers?
>>>>
>>>> Thanks,
>>>> Martin
>>>>
>>>> On Sat, Dec 18, 2010 at 5:44 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>> In your reducer, you can utilize Reporter (getCounter and incrCounter
>>>>> methods) to pass this information between reducers.
>>>>>
>>>>> On Sat, Dec 18, 2010 at 8:04 AM, Martin Becker <_m...@web.de> wrote:
>>>>>>
>>>>>> Hello everbody,
>>>>>>
>>>>>> I am wondering if there is a feature allowing (in my case) reduce
>>>>>> tasks to communicate. For example by some volatile variables at some
>>>>>> centralized point. Or maybe just notify other running or to-be-running
>>>>>> reduce tasks of a completed reduce task featuring some arguments.
>>>>>>
>>>>>> In my case, I have reduce tasks doing computations that will
>>>>>> output/produce certain quality threshold. Other reduce tasks can/could
>>>>>> estimate, if they ever get above those thresholds. If not they could
>>>>>> just cease running.
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Martin
>>>>>>
>>>>>> PS: If such functions are not yet part of the API, I would like to
>>>>>> know if there are good reasons for it and if not, propose to introduce
>>>>>> such functionality.
>>>>>
>>>>>
>>>
>
Re: Passing messages
Posted by Jason <ur...@gmail.com>.
> real time values are not required. Some lagging is tolerable. The
> value/threshold communication is only needed to keep other reducers
> from doing unnecessary work
Then I think counters is what you really need here (assuming they get really updated, but I never tried that)
Sent from my iPhone 4
On Dec 18, 2010, at 3:19 PM, Martin Becker <_m...@web.de> wrote:
> Hello Jason,
>
> real time values are not required. Some lagging is tolerable. The
> value/threshold communication is only needed to keep other reducers
> from doing unnecessary work. Some upper bound would be nice to know,
> though. A single reducer is not an option for my algorithm. That would
> defeat the purpose of using MapReduce. I still would like to know, if
> there is something more general than Counters. Using other data types
> would be convenient. Even better would be, as mentioned, some kind of
> simple message passing system. It seems that neither is supported by
> Hadoop as of now?
>
> Thank you,
> Martin
>
> On Sat, Dec 18, 2010 at 10:07 PM, Jason <ur...@gmail.com> wrote:
>>> Reducers would retrieve that increased value when accessing the same
>>> Counter?
>>
>> I do not think counters reflect real time value. Even if they get updated the values will lag.
>> If you require uptodate value I am afraid you will have to run a single reducer.
>>
>> Sent from my iPhone 4
>>
>> On Dec 18, 2010, at 10:33 AM, Martin Becker <_m...@web.de> wrote:
>>
>>> Thank you Ted,
>>>
>>> I am using the 21.0 API so I would be drawing Counters from the
>>> Context. So if a Counter is increased on a certain Reducer other
>>> Reducers would retrieve that increased value when accessing the same
>>> Counter? If so, then that is an interesting piece of information.
>>>
>>> Unfortunately my threshold are doubles. I guess, I could find some
>>> kind of conversion there. But is there any more general way to pass
>>> information between Reducers?
>>>
>>> Thanks,
>>> Martin
>>>
>>> On Sat, Dec 18, 2010 at 5:44 PM, Ted Yu <yu...@gmail.com> wrote:
>>>> In your reducer, you can utilize Reporter (getCounter and incrCounter
>>>> methods) to pass this information between reducers.
>>>>
>>>> On Sat, Dec 18, 2010 at 8:04 AM, Martin Becker <_m...@web.de> wrote:
>>>>>
>>>>> Hello everbody,
>>>>>
>>>>> I am wondering if there is a feature allowing (in my case) reduce
>>>>> tasks to communicate. For example by some volatile variables at some
>>>>> centralized point. Or maybe just notify other running or to-be-running
>>>>> reduce tasks of a completed reduce task featuring some arguments.
>>>>>
>>>>> In my case, I have reduce tasks doing computations that will
>>>>> output/produce certain quality threshold. Other reduce tasks can/could
>>>>> estimate, if they ever get above those thresholds. If not they could
>>>>> just cease running.
>>>>>
>>>>> Thanks in advance,
>>>>> Martin
>>>>>
>>>>> PS: If such functions are not yet part of the API, I would like to
>>>>> know if there are good reasons for it and if not, propose to introduce
>>>>> such functionality.
>>>>
>>>>
>>
Re: Passing messages
Posted by Martin Becker <_m...@web.de>.
Hello Jason,
real time values are not required. Some lagging is tolerable. The
value/threshold communication is only needed to keep other reducers
from doing unnecessary work. Some upper bound would be nice to know,
though. A single reducer is not an option for my algorithm. That would
defeat the purpose of using MapReduce. I still would like to know, if
there is something more general than Counters. Using other data types
would be convenient. Even better would be, as mentioned, some kind of
simple message passing system. It seems that neither is supported by
Hadoop as of now?
Thank you,
Martin
On Sat, Dec 18, 2010 at 10:07 PM, Jason <ur...@gmail.com> wrote:
>> Reducers would retrieve that increased value when accessing the same
>> Counter?
>
> I do not think counters reflect real time value. Even if they get updated the values will lag.
> If you require uptodate value I am afraid you will have to run a single reducer.
>
> Sent from my iPhone 4
>
> On Dec 18, 2010, at 10:33 AM, Martin Becker <_m...@web.de> wrote:
>
>> Thank you Ted,
>>
>> I am using the 21.0 API so I would be drawing Counters from the
>> Context. So if a Counter is increased on a certain Reducer other
>> Reducers would retrieve that increased value when accessing the same
>> Counter? If so, then that is an interesting piece of information.
>>
>> Unfortunately my threshold are doubles. I guess, I could find some
>> kind of conversion there. But is there any more general way to pass
>> information between Reducers?
>>
>> Thanks,
>> Martin
>>
>> On Sat, Dec 18, 2010 at 5:44 PM, Ted Yu <yu...@gmail.com> wrote:
>>> In your reducer, you can utilize Reporter (getCounter and incrCounter
>>> methods) to pass this information between reducers.
>>>
>>> On Sat, Dec 18, 2010 at 8:04 AM, Martin Becker <_m...@web.de> wrote:
>>>>
>>>> Hello everbody,
>>>>
>>>> I am wondering if there is a feature allowing (in my case) reduce
>>>> tasks to communicate. For example by some volatile variables at some
>>>> centralized point. Or maybe just notify other running or to-be-running
>>>> reduce tasks of a completed reduce task featuring some arguments.
>>>>
>>>> In my case, I have reduce tasks doing computations that will
>>>> output/produce certain quality threshold. Other reduce tasks can/could
>>>> estimate, if they ever get above those thresholds. If not they could
>>>> just cease running.
>>>>
>>>> Thanks in advance,
>>>> Martin
>>>>
>>>> PS: If such functions are not yet part of the API, I would like to
>>>> know if there are good reasons for it and if not, propose to introduce
>>>> such functionality.
>>>
>>>
>
Re: Passing messages
Posted by Jason <ur...@gmail.com>.
> Reducers would retrieve that increased value when accessing the same
> Counter?
I do not think counters reflect real time value. Even if they get updated the values will lag.
If you require uptodate value I am afraid you will have to run a single reducer.
Sent from my iPhone 4
On Dec 18, 2010, at 10:33 AM, Martin Becker <_m...@web.de> wrote:
> Thank you Ted,
>
> I am using the 21.0 API so I would be drawing Counters from the
> Context. So if a Counter is increased on a certain Reducer other
> Reducers would retrieve that increased value when accessing the same
> Counter? If so, then that is an interesting piece of information.
>
> Unfortunately my threshold are doubles. I guess, I could find some
> kind of conversion there. But is there any more general way to pass
> information between Reducers?
>
> Thanks,
> Martin
>
> On Sat, Dec 18, 2010 at 5:44 PM, Ted Yu <yu...@gmail.com> wrote:
>> In your reducer, you can utilize Reporter (getCounter and incrCounter
>> methods) to pass this information between reducers.
>>
>> On Sat, Dec 18, 2010 at 8:04 AM, Martin Becker <_m...@web.de> wrote:
>>>
>>> Hello everbody,
>>>
>>> I am wondering if there is a feature allowing (in my case) reduce
>>> tasks to communicate. For example by some volatile variables at some
>>> centralized point. Or maybe just notify other running or to-be-running
>>> reduce tasks of a completed reduce task featuring some arguments.
>>>
>>> In my case, I have reduce tasks doing computations that will
>>> output/produce certain quality threshold. Other reduce tasks can/could
>>> estimate, if they ever get above those thresholds. If not they could
>>> just cease running.
>>>
>>> Thanks in advance,
>>> Martin
>>>
>>> PS: If such functions are not yet part of the API, I would like to
>>> know if there are good reasons for it and if not, propose to introduce
>>> such functionality.
>>
>>
Re: Passing messages
Posted by Martin Becker <_m...@web.de>.
Thank you Ted,
I am using the 21.0 API so I would be drawing Counters from the
Context. So if a Counter is increased on a certain Reducer other
Reducers would retrieve that increased value when accessing the same
Counter? If so, then that is an interesting piece of information.
Unfortunately my threshold are doubles. I guess, I could find some
kind of conversion there. But is there any more general way to pass
information between Reducers?
Thanks,
Martin
On Sat, Dec 18, 2010 at 5:44 PM, Ted Yu <yu...@gmail.com> wrote:
> In your reducer, you can utilize Reporter (getCounter and incrCounter
> methods) to pass this information between reducers.
>
> On Sat, Dec 18, 2010 at 8:04 AM, Martin Becker <_m...@web.de> wrote:
>>
>> Hello everbody,
>>
>> I am wondering if there is a feature allowing (in my case) reduce
>> tasks to communicate. For example by some volatile variables at some
>> centralized point. Or maybe just notify other running or to-be-running
>> reduce tasks of a completed reduce task featuring some arguments.
>>
>> In my case, I have reduce tasks doing computations that will
>> output/produce certain quality threshold. Other reduce tasks can/could
>> estimate, if they ever get above those thresholds. If not they could
>> just cease running.
>>
>> Thanks in advance,
>> Martin
>>
>> PS: If such functions are not yet part of the API, I would like to
>> know if there are good reasons for it and if not, propose to introduce
>> such functionality.
>
>
Re: Passing messages
Posted by Ted Yu <yu...@gmail.com>.
In your reducer, you can utilize Reporter (getCounter and incrCounter
methods) to pass this information between reducers.
On Sat, Dec 18, 2010 at 8:04 AM, Martin Becker <_m...@web.de> wrote:
> Hello everbody,
>
> I am wondering if there is a feature allowing (in my case) reduce
> tasks to communicate. For example by some volatile variables at some
> centralized point. Or maybe just notify other running or to-be-running
> reduce tasks of a completed reduce task featuring some arguments.
>
> In my case, I have reduce tasks doing computations that will
> output/produce certain quality threshold. Other reduce tasks can/could
> estimate, if they ever get above those thresholds. If not they could
> just cease running.
>
> Thanks in advance,
> Martin
>
> PS: If such functions are not yet part of the API, I would like to
> know if there are good reasons for it and if not, propose to introduce
> such functionality.
>