You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2014/02/10 15:44:43 UTC
very long timeout on failed RM connect
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I've tried setting all of this at once:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
conf.set("ipc.client.connect.max.retries", "2");
and it still seems to wait about 15 seconds.
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Wednesday, February 12, 2014 3:44 PM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> sets the upper layer clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the underlying ipc. In each ClientRMProxy retry, the max number of underlying ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>> wrote:
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net<ma...@redpoint.net>]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I've tried setting all of this at once:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
conf.set("ipc.client.connect.max.retries", "2");
and it still seems to wait about 15 seconds.
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Wednesday, February 12, 2014 3:44 PM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> sets the upper layer clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the underlying ipc. In each ClientRMProxy retry, the max number of underlying ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>> wrote:
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net<ma...@redpoint.net>]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I've tried setting all of this at once:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
conf.set("ipc.client.connect.max.retries", "2");
and it still seems to wait about 15 seconds.
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Wednesday, February 12, 2014 3:44 PM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> sets the upper layer clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the underlying ipc. In each ClientRMProxy retry, the max number of underlying ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>> wrote:
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net<ma...@redpoint.net>]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I've tried setting all of this at once:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
conf.set("ipc.client.connect.max.retries", "2");
and it still seems to wait about 15 seconds.
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Wednesday, February 12, 2014 3:44 PM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> sets the upper layer clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the underlying ipc. In each ClientRMProxy retry, the max number of underlying ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>> wrote:
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net<ma...@redpoint.net>]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms>", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms>", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms sets the upper layer
clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the
underlying ipc. In each ClientRMProxy retry, the max number of underlying
ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Setting
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
>
> still results in a wait of around 15 seconds. Setting this:
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> Also does not help. Is there a retry parameter that can be set?
>
> Thanks
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net]
> *Sent:* Monday, February 10, 2014 12:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: very long timeout on failed RM connect
>
>
>
> I tried:
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
>
>
>
> But it has no apparent effect. Still hangs for a very long time.
>
> john
>
>
>
> *From:* Jian He [mailto:jhe@hortonworks.com <jh...@hortonworks.com>]
> *Sent:* Monday, February 10, 2014 11:05 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: very long timeout on failed RM connect
>
>
>
> Setting the following two properties may solve your problem.
>
> yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
> establish connection to ResourceManager.
>
> yarn.resourcemanager.connect.retry-interval.ms controls How often to try
> connecting to the ResourceManager.
>
>
>
> Jian
>
>
>
> On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>
> wrote:
>
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms sets the upper layer
clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the
underlying ipc. In each ClientRMProxy retry, the max number of underlying
ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Setting
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
>
> still results in a wait of around 15 seconds. Setting this:
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> Also does not help. Is there a retry parameter that can be set?
>
> Thanks
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net]
> *Sent:* Monday, February 10, 2014 12:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: very long timeout on failed RM connect
>
>
>
> I tried:
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
>
>
>
> But it has no apparent effect. Still hangs for a very long time.
>
> john
>
>
>
> *From:* Jian He [mailto:jhe@hortonworks.com <jh...@hortonworks.com>]
> *Sent:* Monday, February 10, 2014 11:05 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: very long timeout on failed RM connect
>
>
>
> Setting the following two properties may solve your problem.
>
> yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
> establish connection to ResourceManager.
>
> yarn.resourcemanager.connect.retry-interval.ms controls How often to try
> connecting to the ResourceManager.
>
>
>
> Jian
>
>
>
> On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>
> wrote:
>
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms sets the upper layer
clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the
underlying ipc. In each ClientRMProxy retry, the max number of underlying
ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Setting
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
>
> still results in a wait of around 15 seconds. Setting this:
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> Also does not help. Is there a retry parameter that can be set?
>
> Thanks
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net]
> *Sent:* Monday, February 10, 2014 12:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: very long timeout on failed RM connect
>
>
>
> I tried:
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
>
>
>
> But it has no apparent effect. Still hangs for a very long time.
>
> john
>
>
>
> *From:* Jian He [mailto:jhe@hortonworks.com <jh...@hortonworks.com>]
> *Sent:* Monday, February 10, 2014 11:05 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: very long timeout on failed RM connect
>
>
>
> Setting the following two properties may solve your problem.
>
> yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
> establish connection to ResourceManager.
>
> yarn.resourcemanager.connect.retry-interval.ms controls How often to try
> connecting to the ResourceManager.
>
>
>
> Jian
>
>
>
> On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>
> wrote:
>
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
ipc.client.connect.retry.interval sets the underlying ipc retry interval
yarn.resourcemanager.connect.retry-interval.ms sets the upper layer
clientRMProxy retry interval.
Each clientRMProxy retry includes one full round of retries of the
underlying ipc. In each ClientRMProxy retry, the max number of underlying
ipc retry is controlled by "ipc.client.connect.max.retries".
Did you try setting both ?
Jian
On Wed, Feb 12, 2014 at 8:36 AM, John Lilley <jo...@redpoint.net>wrote:
> Setting
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
>
> still results in a wait of around 15 seconds. Setting this:
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> Also does not help. Is there a retry parameter that can be set?
>
> Thanks
>
> John
>
>
>
> *From:* John Lilley [mailto:john.lilley@redpoint.net]
> *Sent:* Monday, February 10, 2014 12:12 PM
> *To:* user@hadoop.apache.org
> *Subject:* RE: very long timeout on failed RM connect
>
>
>
> I tried:
>
> conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
>
> conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
>
>
>
> But it has no apparent effect. Still hangs for a very long time.
>
> john
>
>
>
> *From:* Jian He [mailto:jhe@hortonworks.com <jh...@hortonworks.com>]
> *Sent:* Monday, February 10, 2014 11:05 AM
> *To:* user@hadoop.apache.org
> *Subject:* Re: very long timeout on failed RM connect
>
>
>
> Setting the following two properties may solve your problem.
>
> yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
> establish connection to ResourceManager.
>
> yarn.resourcemanager.connect.retry-interval.ms controls How often to try
> connecting to the ResourceManager.
>
>
>
> Jian
>
>
>
> On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>
> wrote:
>
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
Setting
conf.set("yarn.resourcemanager.connect.max-wait.ms", "500");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "500");
still results in a wait of around 15 seconds. Setting this:
conf.set("ipc.client.connect.max.retries", "2");
Also does not help. Is there a retry parameter that can be set?
Thanks
John
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Monday, February 10, 2014 12:12 PM
To: user@hadoop.apache.org
Subject: RE: very long timeout on failed RM connect
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: very long timeout on failed RM connect
Posted by John Lilley <jo...@redpoint.net>.
I tried:
conf.set("yarn.resourcemanager.connect.max-wait.ms", "10000");
conf.set("yarn.resourcemanager.connect.retry-interval.ms", "1000");
But it has no apparent effect. Still hangs for a very long time.
john
From: Jian He [mailto:jhe@hortonworks.com]
Sent: Monday, February 10, 2014 11:05 AM
To: user@hadoop.apache.org
Subject: Re: very long timeout on failed RM connect
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms<http://yarn.resourcemanager.connect.max-wait.ms/> controls Maximum time to wait to establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms<http://yarn.resourcemanager.connect.retry-interval.ms/> controls How often to try connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>> wrote:
Our application (running outside the Hadoop cluster) connects to the RM through YarnClient. This works fine, except we've found that if the RM address or port is misconfigured in our software, or a firewall blocks access, the first call into the client (in this case getNodeReports) hangs for a very long time. I've tried
conf.set("ipc.client.connect.max.retries", "2");
But this doesn't help. Is there a configuration setting I can make on the YarnClient that will reduce this hang time?
I understand why this long-winded retry strategy exists, in order to prevent a highly-loaded cluster from failing jobs. But it is not appropriate for an interactive application.
Thanks
John
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms controls How often to try
connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>wrote:
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms controls How often to try
connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>wrote:
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms controls How often to try
connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>wrote:
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: very long timeout on failed RM connect
Posted by Jian He <jh...@hortonworks.com>.
Setting the following two properties may solve your problem.
yarn.resourcemanager.connect.max-wait.ms controls Maximum time to wait to
establish connection to ResourceManager.
yarn.resourcemanager.connect.retry-interval.ms controls How often to try
connecting to the ResourceManager.
Jian
On Mon, Feb 10, 2014 at 6:44 AM, John Lilley <jo...@redpoint.net>wrote:
> Our application (running outside the Hadoop cluster) connects to the RM
> through YarnClient. This works fine, except we've found that if the RM
> address or port is misconfigured in our software, or a firewall blocks
> access, the first call into the client (in this case getNodeReports) hangs
> for a very long time. I've tried
>
> conf.set("ipc.client.connect.max.retries", "2");
>
> But this doesn't help. Is there a configuration setting I can make on the
> YarnClient that will reduce this hang time?
>
> I understand why this long-winded retry strategy exists, in order to
> prevent a highly-loaded cluster from failing jobs. But it is not
> appropriate for an interactive application.
>
> Thanks
>
> John
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.