You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by blah blah <tm...@gmail.com> on 2013/03/27 13:08:12 UTC
Yarn communication between containers
Hi
I would like the containers to communicate with each other (also some heart
beats to AppMaster). How can I achieve this? Is there some code snippet in
the Yarn project or tutorial, that could guide me on implementing this
functionality?
regards
tmp
Re: Yarn communication between containers
Posted by blah blah <tm...@gmail.com>.
Hi Sandy
I noticed that Yarn uses Jetty, so I thought maybe there is some API hidden
to provide such a functionality (for MapReduce implementation). My current
idea is to start netty server/client on each container. I will definitely
look up the classes you mentioned. Thanks for code guidance (Yarn is a big
project and sometimes grep is not enough ;) )
regards
tmp
2013/3/27 Sandy Ryza <sa...@cloudera.com>
> Hi tmp,
>
> YARN doesn't provide an explicit protocol for doing this. Applications
> are expected to have their own mechanism for communication between task
> containers, other task containers, and app masters. If you want to see how
> this is done in MapReduce, I would suggest looking at the
> TaskUmbilicalProtocol, which is implemented on the AM side by
> TaskAttemptListenerImpl and created on the Container side in YarnChild.
> Through such a protocol, you could give containers the locations of other
> containers that would allow them to communicate correctly.
>
> -Sandy
>
>
> On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
>
>> Hi
>>
>> I would like the containers to communicate with each other (also some
>> heart beats to AppMaster). How can I achieve this? Is there some code
>> snippet in the Yarn project or tutorial, that could guide me on
>> implementing this functionality?
>>
>> regards
>> tmp
>>
>
>
Re: Yarn communication between containers
Posted by blah blah <tm...@gmail.com>.
Hi Sandy
I noticed that Yarn uses Jetty, so I thought maybe there is some API hidden
to provide such a functionality (for MapReduce implementation). My current
idea is to start netty server/client on each container. I will definitely
look up the classes you mentioned. Thanks for code guidance (Yarn is a big
project and sometimes grep is not enough ;) )
regards
tmp
2013/3/27 Sandy Ryza <sa...@cloudera.com>
> Hi tmp,
>
> YARN doesn't provide an explicit protocol for doing this. Applications
> are expected to have their own mechanism for communication between task
> containers, other task containers, and app masters. If you want to see how
> this is done in MapReduce, I would suggest looking at the
> TaskUmbilicalProtocol, which is implemented on the AM side by
> TaskAttemptListenerImpl and created on the Container side in YarnChild.
> Through such a protocol, you could give containers the locations of other
> containers that would allow them to communicate correctly.
>
> -Sandy
>
>
> On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
>
>> Hi
>>
>> I would like the containers to communicate with each other (also some
>> heart beats to AppMaster). How can I achieve this? Is there some code
>> snippet in the Yarn project or tutorial, that could guide me on
>> implementing this functionality?
>>
>> regards
>> tmp
>>
>
>
Re: Yarn communication between containers
Posted by blah blah <tm...@gmail.com>.
Hi Sandy
I noticed that Yarn uses Jetty, so I thought maybe there is some API hidden
to provide such a functionality (for MapReduce implementation). My current
idea is to start netty server/client on each container. I will definitely
look up the classes you mentioned. Thanks for code guidance (Yarn is a big
project and sometimes grep is not enough ;) )
regards
tmp
2013/3/27 Sandy Ryza <sa...@cloudera.com>
> Hi tmp,
>
> YARN doesn't provide an explicit protocol for doing this. Applications
> are expected to have their own mechanism for communication between task
> containers, other task containers, and app masters. If you want to see how
> this is done in MapReduce, I would suggest looking at the
> TaskUmbilicalProtocol, which is implemented on the AM side by
> TaskAttemptListenerImpl and created on the Container side in YarnChild.
> Through such a protocol, you could give containers the locations of other
> containers that would allow them to communicate correctly.
>
> -Sandy
>
>
> On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
>
>> Hi
>>
>> I would like the containers to communicate with each other (also some
>> heart beats to AppMaster). How can I achieve this? Is there some code
>> snippet in the Yarn project or tutorial, that could guide me on
>> implementing this functionality?
>>
>> regards
>> tmp
>>
>
>
Re: Yarn communication between containers
Posted by blah blah <tm...@gmail.com>.
Hi Sandy
I noticed that Yarn uses Jetty, so I thought maybe there is some API hidden
to provide such a functionality (for MapReduce implementation). My current
idea is to start netty server/client on each container. I will definitely
look up the classes you mentioned. Thanks for code guidance (Yarn is a big
project and sometimes grep is not enough ;) )
regards
tmp
2013/3/27 Sandy Ryza <sa...@cloudera.com>
> Hi tmp,
>
> YARN doesn't provide an explicit protocol for doing this. Applications
> are expected to have their own mechanism for communication between task
> containers, other task containers, and app masters. If you want to see how
> this is done in MapReduce, I would suggest looking at the
> TaskUmbilicalProtocol, which is implemented on the AM side by
> TaskAttemptListenerImpl and created on the Container side in YarnChild.
> Through such a protocol, you could give containers the locations of other
> containers that would allow them to communicate correctly.
>
> -Sandy
>
>
> On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
>
>> Hi
>>
>> I would like the containers to communicate with each other (also some
>> heart beats to AppMaster). How can I achieve this? Is there some code
>> snippet in the Yarn project or tutorial, that could guide me on
>> implementing this functionality?
>>
>> regards
>> tmp
>>
>
>
Re: Yarn communication between containers
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi tmp,
YARN doesn't provide an explicit protocol for doing this. Applications are
expected to have their own mechanism for communication between task
containers, other task containers, and app masters. If you want to see how
this is done in MapReduce, I would suggest looking at the
TaskUmbilicalProtocol, which is implemented on the AM side by
TaskAttemptListenerImpl and created on the Container side in YarnChild.
Through such a protocol, you could give containers the locations of other
containers that would allow them to communicate correctly.
-Sandy
On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
> Hi
>
> I would like the containers to communicate with each other (also some
> heart beats to AppMaster). How can I achieve this? Is there some code
> snippet in the Yarn project or tutorial, that could guide me on
> implementing this functionality?
>
> regards
> tmp
>
Re: Yarn communication between containers
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi tmp,
YARN doesn't provide an explicit protocol for doing this. Applications are
expected to have their own mechanism for communication between task
containers, other task containers, and app masters. If you want to see how
this is done in MapReduce, I would suggest looking at the
TaskUmbilicalProtocol, which is implemented on the AM side by
TaskAttemptListenerImpl and created on the Container side in YarnChild.
Through such a protocol, you could give containers the locations of other
containers that would allow them to communicate correctly.
-Sandy
On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
> Hi
>
> I would like the containers to communicate with each other (also some
> heart beats to AppMaster). How can I achieve this? Is there some code
> snippet in the Yarn project or tutorial, that could guide me on
> implementing this functionality?
>
> regards
> tmp
>
Re: Yarn communication between containers
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi tmp,
YARN doesn't provide an explicit protocol for doing this. Applications are
expected to have their own mechanism for communication between task
containers, other task containers, and app masters. If you want to see how
this is done in MapReduce, I would suggest looking at the
TaskUmbilicalProtocol, which is implemented on the AM side by
TaskAttemptListenerImpl and created on the Container side in YarnChild.
Through such a protocol, you could give containers the locations of other
containers that would allow them to communicate correctly.
-Sandy
On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
> Hi
>
> I would like the containers to communicate with each other (also some
> heart beats to AppMaster). How can I achieve this? Is there some code
> snippet in the Yarn project or tutorial, that could guide me on
> implementing this functionality?
>
> regards
> tmp
>
Re: Yarn communication between containers
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi tmp,
YARN doesn't provide an explicit protocol for doing this. Applications are
expected to have their own mechanism for communication between task
containers, other task containers, and app masters. If you want to see how
this is done in MapReduce, I would suggest looking at the
TaskUmbilicalProtocol, which is implemented on the AM side by
TaskAttemptListenerImpl and created on the Container side in YarnChild.
Through such a protocol, you could give containers the locations of other
containers that would allow them to communicate correctly.
-Sandy
On Wed, Mar 27, 2013 at 5:08 AM, blah blah <tm...@gmail.com> wrote:
> Hi
>
> I would like the containers to communicate with each other (also some
> heart beats to AppMaster). How can I achieve this? Is there some code
> snippet in the Yarn project or tutorial, that could guide me on
> implementing this functionality?
>
> regards
> tmp
>