You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Chris Mawata <ch...@gmail.com> on 2015/03/28 20:40:27 UTC
Does Hadoop 2.6.0 have job level blacklisting?
At
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
is a description of how you can have a script check the health of a node
and indicate to the ResourceManager that it is unhealthy. This seems to be
at the cluster level. Is there still job level blacklisting as there was in
earlier versions?
Chris Mawata
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Chris Mawata <ch...@gmail.com>.
Thanks very much Rohith.
Chris
On Mar 30, 2015 12:11 AM, "Rohith Sharma K S" <ro...@huawei.com>
wrote:
> Hi Chris
>
>
>
> Is there still job level blacklisting as there was in earlier versions?
>
> >> yes, job level blacklisting support is there. Application Master has
> to identify the nodes which it wants to blacklists and send those nodes
> details to ResourceManager via ApplicationMasterProtocol#allocate request.
> On blacklisted nodes, containers will not be assigned thereafter.
>
>
>
> * Java Doc*
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:* Chris Mawata [mailto:chris.mawata@gmail.com]
> *Sent:* 29 March 2015 01:10
> *To:* user@hadoop.apache.org
> *Subject:* Does Hadoop 2.6.0 have job level blacklisting?
>
>
>
> At
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
>
> is a description of how you can have a script check the health of a node
> and indicate to the ResourceManager that it is unhealthy. This seems to be
> at the cluster level. Is there still job level blacklisting as there was in
> earlier versions?
>
>
>
> Chris Mawata
>
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Chris Mawata <ch...@gmail.com>.
Thanks very much Rohith.
Chris
On Mar 30, 2015 12:11 AM, "Rohith Sharma K S" <ro...@huawei.com>
wrote:
> Hi Chris
>
>
>
> Is there still job level blacklisting as there was in earlier versions?
>
> >> yes, job level blacklisting support is there. Application Master has
> to identify the nodes which it wants to blacklists and send those nodes
> details to ResourceManager via ApplicationMasterProtocol#allocate request.
> On blacklisted nodes, containers will not be assigned thereafter.
>
>
>
> * Java Doc*
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:* Chris Mawata [mailto:chris.mawata@gmail.com]
> *Sent:* 29 March 2015 01:10
> *To:* user@hadoop.apache.org
> *Subject:* Does Hadoop 2.6.0 have job level blacklisting?
>
>
>
> At
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
>
> is a description of how you can have a script check the health of a node
> and indicate to the ResourceManager that it is unhealthy. This seems to be
> at the cluster level. Is there still job level blacklisting as there was in
> earlier versions?
>
>
>
> Chris Mawata
>
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Chris Mawata <ch...@gmail.com>.
Thanks very much Rohith.
Chris
On Mar 30, 2015 12:11 AM, "Rohith Sharma K S" <ro...@huawei.com>
wrote:
> Hi Chris
>
>
>
> Is there still job level blacklisting as there was in earlier versions?
>
> >> yes, job level blacklisting support is there. Application Master has
> to identify the nodes which it wants to blacklists and send those nodes
> details to ResourceManager via ApplicationMasterProtocol#allocate request.
> On blacklisted nodes, containers will not be assigned thereafter.
>
>
>
> * Java Doc*
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:* Chris Mawata [mailto:chris.mawata@gmail.com]
> *Sent:* 29 March 2015 01:10
> *To:* user@hadoop.apache.org
> *Subject:* Does Hadoop 2.6.0 have job level blacklisting?
>
>
>
> At
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
>
> is a description of how you can have a script check the health of a node
> and indicate to the ResourceManager that it is unhealthy. This seems to be
> at the cluster level. Is there still job level blacklisting as there was in
> earlier versions?
>
>
>
> Chris Mawata
>
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Chris Mawata <ch...@gmail.com>.
Thanks very much Rohith.
Chris
On Mar 30, 2015 12:11 AM, "Rohith Sharma K S" <ro...@huawei.com>
wrote:
> Hi Chris
>
>
>
> Is there still job level blacklisting as there was in earlier versions?
>
> >> yes, job level blacklisting support is there. Application Master has
> to identify the nodes which it wants to blacklists and send those nodes
> details to ResourceManager via ApplicationMasterProtocol#allocate request.
> On blacklisted nodes, containers will not be assigned thereafter.
>
>
>
> * Java Doc*
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
>
>
> https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
>
>
>
> Thanks & Regards
>
> Rohith Sharma K S
>
> *From:* Chris Mawata [mailto:chris.mawata@gmail.com]
> *Sent:* 29 March 2015 01:10
> *To:* user@hadoop.apache.org
> *Subject:* Does Hadoop 2.6.0 have job level blacklisting?
>
>
>
> At
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
>
> is a description of how you can have a script check the health of a node
> and indicate to the ResourceManager that it is unhealthy. This seems to be
> at the cluster level. Is there still job level blacklisting as there was in
> earlier versions?
>
>
>
> Chris Mawata
>
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Chris
Is there still job level blacklisting as there was in earlier versions?
>> yes, job level blacklisting support is there. Application Master has to identify the nodes which it wants to blacklists and send those nodes details to ResourceManager via ApplicationMasterProtocol#allocate request. On blacklisted nodes, containers will not be assigned thereafter.
Java Doc
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
Thanks & Regards
Rohith Sharma K S
From: Chris Mawata [mailto:chris.mawata@gmail.com]
Sent: 29 March 2015 01:10
To: user@hadoop.apache.org
Subject: Does Hadoop 2.6.0 have job level blacklisting?
At http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
is a description of how you can have a script check the health of a node and indicate to the ResourceManager that it is unhealthy. This seems to be at the cluster level. Is there still job level blacklisting as there was in earlier versions?
Chris Mawata
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Chris
Is there still job level blacklisting as there was in earlier versions?
>> yes, job level blacklisting support is there. Application Master has to identify the nodes which it wants to blacklists and send those nodes details to ResourceManager via ApplicationMasterProtocol#allocate request. On blacklisted nodes, containers will not be assigned thereafter.
Java Doc
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
Thanks & Regards
Rohith Sharma K S
From: Chris Mawata [mailto:chris.mawata@gmail.com]
Sent: 29 March 2015 01:10
To: user@hadoop.apache.org
Subject: Does Hadoop 2.6.0 have job level blacklisting?
At http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
is a description of how you can have a script check the health of a node and indicate to the ResourceManager that it is unhealthy. This seems to be at the cluster level. Is there still job level blacklisting as there was in earlier versions?
Chris Mawata
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Chris
Is there still job level blacklisting as there was in earlier versions?
>> yes, job level blacklisting support is there. Application Master has to identify the nodes which it wants to blacklists and send those nodes details to ResourceManager via ApplicationMasterProtocol#allocate request. On blacklisted nodes, containers will not be assigned thereafter.
Java Doc
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
Thanks & Regards
Rohith Sharma K S
From: Chris Mawata [mailto:chris.mawata@gmail.com]
Sent: 29 March 2015 01:10
To: user@hadoop.apache.org
Subject: Does Hadoop 2.6.0 have job level blacklisting?
At http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
is a description of how you can have a script check the health of a node and indicate to the ResourceManager that it is unhealthy. This seems to be at the cluster level. Is there still job level blacklisting as there was in earlier versions?
Chris Mawata
RE: Does Hadoop 2.6.0 have job level blacklisting?
Posted by Rohith Sharma K S <ro...@huawei.com>.
Hi Chris
Is there still job level blacklisting as there was in earlier versions?
>> yes, job level blacklisting support is there. Application Master has to identify the nodes which it wants to blacklists and send those nodes details to ResourceManager via ApplicationMasterProtocol#allocate request. On blacklisted nodes, containers will not be assigned thereafter.
Java Doc
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/ApplicationMasterProtocol.html#allocate(org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest)
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/protocolrecords/AllocateRequest.html
Thanks & Regards
Rohith Sharma K S
From: Chris Mawata [mailto:chris.mawata@gmail.com]
Sent: 29 March 2015 01:10
To: user@hadoop.apache.org
Subject: Does Hadoop 2.6.0 have job level blacklisting?
At http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html#Monitoring_Health_of_NodeManagers
is a description of how you can have a script check the health of a node and indicate to the ResourceManager that it is unhealthy. This seems to be at the cluster level. Is there still job level blacklisting as there was in earlier versions?
Chris Mawata