You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2014/01/11 20:37:06 UTC
How to make AM terminate if client crashes?
We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
Thanks
John
Re: How to make AM terminate if client crashes?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
You can set the maximum map and reduce attempts so that if a failure
occurred job gets failed and done.
On Sat, Jan 11, 2014 at 11:07 PM, John Lilley <jo...@redpoint.net>wrote:
> We have a YARN application that we want to automatically terminate if
> the YARN client disconnects or crashes. Is it possible to configure the
> YarnClient-RM connection so that if the client terminates the RM
> automatically terminates the AM? Or do we need to build our own logic
> (e.g. a direct client-AM connection) for that?
>
> Thanks
>
> John
>
Re: How to make AM terminate if client crashes?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
You can set the maximum map and reduce attempts so that if a failure
occurred job gets failed and done.
On Sat, Jan 11, 2014 at 11:07 PM, John Lilley <jo...@redpoint.net>wrote:
> We have a YARN application that we want to automatically terminate if
> the YARN client disconnects or crashes. Is it possible to configure the
> YarnClient-RM connection so that if the client terminates the RM
> automatically terminates the AM? Or do we need to build our own logic
> (e.g. a direct client-AM connection) for that?
>
> Thanks
>
> John
>
Re: How to make AM terminate if client crashes?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
You can set the maximum map and reduce attempts so that if a failure
occurred job gets failed and done.
On Sat, Jan 11, 2014 at 11:07 PM, John Lilley <jo...@redpoint.net>wrote:
> We have a YARN application that we want to automatically terminate if
> the YARN client disconnects or crashes. Is it possible to configure the
> YarnClient-RM connection so that if the client terminates the RM
> automatically terminates the AM? Or do we need to build our own logic
> (e.g. a direct client-AM connection) for that?
>
> Thanks
>
> John
>
Re: How to make AM terminate if client crashes?
Posted by Hitesh Shah <hi...@apache.org>.
You would probably need to bake this into your own application. By default, a client never should need to keep an open active connection with the RM. It could keep an active connection with the AM ( application-specific code required ) but it would then also have to handle failover to a different AM if the first AM crashes and a new attempt is launched by the RM.
Does your application have access to a zookeeper cluster within your installation? It would be simple enough for someone to build a library which either uses an ephemeral node or updates some form of an applicationId specific keep-alive flag that the AM can monitor to make its decisions.
@Vinod, on a different note, does it make sense for the RM to run an embedded ZK to aid fencing across multiple AM attempts which could then be used by the above to piggyback on?
-- Hitesh
On Jan 13, 2014, at 8:30 AM, Vinod Kumar Vavilapalli wrote:
> The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
>
> Any more details on why this is desired?
>
> +Vinod
>
> On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
>
>> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
>> Thanks
>> John
Re: How to make AM terminate if client crashes?
Posted by Hitesh Shah <hi...@apache.org>.
You would probably need to bake this into your own application. By default, a client never should need to keep an open active connection with the RM. It could keep an active connection with the AM ( application-specific code required ) but it would then also have to handle failover to a different AM if the first AM crashes and a new attempt is launched by the RM.
Does your application have access to a zookeeper cluster within your installation? It would be simple enough for someone to build a library which either uses an ephemeral node or updates some form of an applicationId specific keep-alive flag that the AM can monitor to make its decisions.
@Vinod, on a different note, does it make sense for the RM to run an embedded ZK to aid fencing across multiple AM attempts which could then be used by the above to piggyback on?
-- Hitesh
On Jan 13, 2014, at 8:30 AM, Vinod Kumar Vavilapalli wrote:
> The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
>
> Any more details on why this is desired?
>
> +Vinod
>
> On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
>
>> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
>> Thanks
>> John
RE: How to make AM terminate if client crashes?
Posted by John Lilley <jo...@redpoint.net>.
In our application the "client" is actually one step of many in an application. If the "client" goes away entirely, that means the entire application has crashed or has been terminated somehow. Our application doesn't checkpoint state at this step, so the step will have to be restarted anyway. This doesn't break anything, it is just a waste to let it keep running. Thanks for clarifying the stateless nature of the client connection.
John
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, January 13, 2014 9:31 AM
To: user@hadoop.apache.org
Subject: Re: How to make AM terminate if client crashes?
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: How to make AM terminate if client crashes?
Posted by John Lilley <jo...@redpoint.net>.
In our application the "client" is actually one step of many in an application. If the "client" goes away entirely, that means the entire application has crashed or has been terminated somehow. Our application doesn't checkpoint state at this step, so the step will have to be restarted anyway. This doesn't break anything, it is just a waste to let it keep running. Thanks for clarifying the stateless nature of the client connection.
John
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, January 13, 2014 9:31 AM
To: user@hadoop.apache.org
Subject: Re: How to make AM terminate if client crashes?
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: How to make AM terminate if client crashes?
Posted by Hitesh Shah <hi...@apache.org>.
You would probably need to bake this into your own application. By default, a client never should need to keep an open active connection with the RM. It could keep an active connection with the AM ( application-specific code required ) but it would then also have to handle failover to a different AM if the first AM crashes and a new attempt is launched by the RM.
Does your application have access to a zookeeper cluster within your installation? It would be simple enough for someone to build a library which either uses an ephemeral node or updates some form of an applicationId specific keep-alive flag that the AM can monitor to make its decisions.
@Vinod, on a different note, does it make sense for the RM to run an embedded ZK to aid fencing across multiple AM attempts which could then be used by the above to piggyback on?
-- Hitesh
On Jan 13, 2014, at 8:30 AM, Vinod Kumar Vavilapalli wrote:
> The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
>
> Any more details on why this is desired?
>
> +Vinod
>
> On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
>
>> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
>> Thanks
>> John
RE: How to make AM terminate if client crashes?
Posted by John Lilley <jo...@redpoint.net>.
In our application the "client" is actually one step of many in an application. If the "client" goes away entirely, that means the entire application has crashed or has been terminated somehow. Our application doesn't checkpoint state at this step, so the step will have to be restarted anyway. This doesn't break anything, it is just a waste to let it keep running. Thanks for clarifying the stateless nature of the client connection.
John
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, January 13, 2014 9:31 AM
To: user@hadoop.apache.org
Subject: Re: How to make AM terminate if client crashes?
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: How to make AM terminate if client crashes?
Posted by John Lilley <jo...@redpoint.net>.
In our application the "client" is actually one step of many in an application. If the "client" goes away entirely, that means the entire application has crashed or has been terminated somehow. Our application doesn't checkpoint state at this step, so the step will have to be restarted anyway. This doesn't break anything, it is just a waste to let it keep running. Thanks for clarifying the stateless nature of the client connection.
John
From: Vinod Kumar Vavilapalli [mailto:vinodkv@hortonworks.com]
Sent: Monday, January 13, 2014 9:31 AM
To: user@hadoop.apache.org
Subject: Re: How to make AM terminate if client crashes?
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: How to make AM terminate if client crashes?
Posted by Hitesh Shah <hi...@apache.org>.
You would probably need to bake this into your own application. By default, a client never should need to keep an open active connection with the RM. It could keep an active connection with the AM ( application-specific code required ) but it would then also have to handle failover to a different AM if the first AM crashes and a new attempt is launched by the RM.
Does your application have access to a zookeeper cluster within your installation? It would be simple enough for someone to build a library which either uses an ephemeral node or updates some form of an applicationId specific keep-alive flag that the AM can monitor to make its decisions.
@Vinod, on a different note, does it make sense for the RM to run an embedded ZK to aid fencing across multiple AM attempts which could then be used by the above to piggyback on?
-- Hitesh
On Jan 13, 2014, at 8:30 AM, Vinod Kumar Vavilapalli wrote:
> The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
>
> Any more details on why this is desired?
>
> +Vinod
>
> On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
>
>> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
>> Thanks
>> John
Re: How to make AM terminate if client crashes?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
> Thanks
> John
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How to make AM terminate if client crashes?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
> Thanks
> John
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How to make AM terminate if client crashes?
Posted by Saeed Adel Mehraban <s....@gmail.com>.
You can set the maximum map and reduce attempts so that if a failure
occurred job gets failed and done.
On Sat, Jan 11, 2014 at 11:07 PM, John Lilley <jo...@redpoint.net>wrote:
> We have a YARN application that we want to automatically terminate if
> the YARN client disconnects or crashes. Is it possible to configure the
> YarnClient-RM connection so that if the client terminates the RM
> automatically terminates the AM? Or do we need to build our own logic
> (e.g. a direct client-AM connection) for that?
>
> Thanks
>
> John
>
Re: How to make AM terminate if client crashes?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
> Thanks
> John
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How to make AM terminate if client crashes?
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
The architecture is built around detachable clients. So, no, it doesn't happen automatically. Even if we were to add that feature, it'd be fraught with edge cases - network issues causing app-termination even though client is still alive etc.
Any more details on why this is desired?
+Vinod
On Jan 11, 2014, at 11:37 AM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN application that we want to automatically terminate if the YARN client disconnects or crashes. Is it possible to configure the YarnClient-RM connection so that if the client terminates the RM automatically terminates the AM? Or do we need to build our own logic (e.g. a direct client-AM connection) for that?
> Thanks
> John
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.