You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Muthupandiyan, Kamaraj" <ka...@hpe.com> on 2020/08/21 14:04:52 UTC
setting up Load balancer for name nodes
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
RE: setting up Load balancer for name nodes
Posted by "Muthupandiyan, Kamaraj" <ka...@hpe.com>.
Hi Mingliang Liu,
Below is the message while I am starting the datanodes from name nodes. For start Datanode, its throwing the below error and for stop its simply exiting( Not even logging is happening)
hdfs@sym-hdfsnn1 bin]$ hdfs --daemon start datanode
ERROR: Cannot set priority of datanode process 94884
[hdfs@sym-hdfsnn1 bin]$ hdfs --daemon stop datanode
In Hadoop version 2.9.2 We used the below command to start the Datanode from namenode and its was working. Post upgrade, we are not able to start the datanodes from name nodes, do we have any other option to start and stop the datanodes from name nodes.
hadoop-daemondaemons.sh --config $HADOOP_CONF_DIR --script hdfs start datanode
Regards,
Kamaraj
InfoSight - SRE
From: Mingliang Liu<ma...@gmail.com>
Sent: Tuesday, August 25, 2020 12:49 PM
To: Muthupandiyan, Kamaraj<ma...@hpe.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: setting up Load balancer for name nodes
Kamaraj,
The manual is usually helpful for basic usages: https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/ClusterSetup.html<https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/ClusterSetup.html>
1. The log you showed did not show "while starting the Datanode...is not working". The command I saw was "hdfs --daemon stop datanode", which is to stop the datanode.
2. The `hdfs --daemon start/stop datanode` is usually issued from the DataNode host. If you need to start all services of an HDFS cluster, try $HADOOP_HOME/sbin/start-dfs.sh
Anyway, I do not see this is related to Hadoop versions.
On Mon, Aug 24, 2020 at 4:29 AM Muthupandiyan, Kamaraj <ka...@hpe.com>> wrote:
Hi Team/Mingliang
We have upgraded to Hadoop 3.2.1 and we are having following issues , can somebody please help me on this.
We have successfully started name node and journal-node. However, while starting the Datanode from namenode is not working , the command simply exits.
[hdfs@sym-hdfsnn2 hadoop]$ hdfs version
Hadoop 3.2.1
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git<https://gitbox.apache.org/repos/asf/hadoop.git> -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842
Compiled by rohithsharmaks on 2019-09-10T15:56Z
Compiled with protoc 2.5.0
From source with checksum 776eaf9eee9c0ffc370bcbc1888737
This command was run using /home/hdfs/hadoop/share/hadoop/common/hadoop-common-3.2.1.jar
[hdfs@sym-hdfsnn2 hadoop]$ hdfs --daemon stop datanode
Regards,
Kamaraj
From: Muthupandiyan, Kamaraj<ma...@hpe.com>
Sent: Sunday, August 23, 2020 9:22 PM
To: Mingliang Liu<ma...@gmail.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: setting up Load balancer for name nodes
Hi Mingliang Liu,
Could you please help me with my additional questions
1. Do we have any other options for setting up the Load balancer for name node in Hadoop Version 2.9.2 ?
1. I got the following document for upgrading the HDFS cluster - https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html<https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html> or Do we have any other document for upgrading Version 2.9.2 to Version 3.2.x
Regards,
Kamaraj
InfoSight - SRE
From: Muthupandiyan, Kamaraj<ma...@hpe.com>
Sent: Sunday, August 23, 2020 6:57 PM
Subject: RE: setting up Load balancer for name nodes
Hi Mingliang Liu,
Thanks a lot for your response, I have read the document for Hadoop 3.3.0 and found that Load balancers will be supported on AWS/Azure, will it supported on A10 Loadbalancers or nginx ?
Regards,
Kamaraj
InfoSight - SRE
From: Mingliang Liu<ma...@gmail.com>
Sent: Saturday, August 22, 2020 2:55 AM
To: Muthupandiyan, Kamaraj<ma...@hpe.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: setting up Load balancer for name nodes
Hi Kamaraj,
I don't think the feature you are referring to exists in Hadoop 2.9.2. The feature was brought up by JIRA https://issues.apache.org/jira/browse/HADOOP-15707<https://issues.apache.org/jira/browse/HADOOP-15707> which is released in Hadoop 3.2+ (see "Fixed Version")
Also, you may want to go through the Hadoop manual/doc for your current Hadoop version. If you change r3.3.0 to r2.9.2 in the URL you posted, you should be reading: https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html>
Thanks,
On Fri, Aug 21, 2020 at 11:02 AM Muthupandiyan, Kamaraj <ka...@hpe.com>> wrote:
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html> and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
--
L
--
L
Re: setting up Load balancer for name nodes
Posted by Mingliang Liu <li...@gmail.com>.
Kamaraj,
The manual is usually helpful for basic usages:
https://hadoop.apache.org/docs/r3.2.1/hadoop-project-dist/hadoop-common/ClusterSetup.html
1. The log you showed did not show "while starting the Datanode...is not
working". The command I saw was "hdfs --daemon stop datanode", which is to
stop the datanode.
2. The `hdfs --daemon start/stop datanode` is usually issued from the
DataNode host. If you need to start all services of an HDFS cluster, try
$HADOOP_HOME/sbin/start-dfs.sh
Anyway, I do not see this is related to Hadoop versions.
On Mon, Aug 24, 2020 at 4:29 AM Muthupandiyan, Kamaraj <
kamaraj.muthupandiyan@hpe.com> wrote:
> Hi Team/Mingliang
>
>
>
> *We have upgraded to Hadoop 3.2.1 and we are having following issues , can
> somebody please help me on this.*
>
>
>
> *We have successfully started name node and journal-node. However, while
> starting the Datanode from namenode is not working , the command simply
> exits.*
>
>
>
> [hdfs@sym-hdfsnn2 hadoop]$ hdfs version
>
> Hadoop 3.2.1
>
> Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r
> b3cbbb467e22ea829b3808f4b7b01d07e0bf3842
>
> Compiled by rohithsharmaks on 2019-09-10T15:56Z
>
> Compiled with protoc 2.5.0
>
> From source with checksum 776eaf9eee9c0ffc370bcbc1888737
>
> This command was run using
> /home/hdfs/hadoop/share/hadoop/common/hadoop-common-3.2.1.jar
>
> [hdfs@sym-hdfsnn2 hadoop]$ hdfs --daemon stop datanode
>
>
>
> Regards,
>
> Kamaraj
>
>
>
>
>
> *From: *Muthupandiyan, Kamaraj <ka...@hpe.com>
> *Sent: *Sunday, August 23, 2020 9:22 PM
> *To: *Mingliang Liu <li...@gmail.com>
> *Cc: *user@hadoop.apache.org
> *Subject: *RE: setting up Load balancer for name nodes
>
>
>
> Hi Mingliang Liu,
>
>
>
> Could you please help me with my additional questions
>
>
>
> 1. Do we have any other options for setting up the Load balancer for
> name node in Hadoop Version 2.9.2 ?
>
>
>
> 1. I got the following document for upgrading the HDFS cluster -
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html
> or Do we have any other document for upgrading Version 2.9.2 to Version
> 3.2.x
>
>
>
>
>
>
>
> Regards,
>
> Kamaraj
>
> InfoSight - SRE
>
>
>
> *From: *Muthupandiyan, Kamaraj <ka...@hpe.com>
> *Sent: *Sunday, August 23, 2020 6:57 PM
> *Subject: *RE: setting up Load balancer for name nodes
>
>
>
> Hi Mingliang Liu,
>
>
>
> Thanks a lot for your response, I have read the document for Hadoop 3.3.0
> and found that Load balancers will be supported on AWS/Azure, will it
> supported on A10 Loadbalancers or nginx ?
>
>
>
> Regards,
>
> Kamaraj
>
> InfoSight - SRE
>
>
>
> *From: *Mingliang Liu <li...@gmail.com>
> *Sent: *Saturday, August 22, 2020 2:55 AM
> *To: *Muthupandiyan, Kamaraj <ka...@hpe.com>
> *Cc: *user@hadoop.apache.org
> *Subject: *Re: setting up Load balancer for name nodes
>
>
>
> Hi Kamaraj,
>
>
>
> I don't think the feature you are referring to exists in Hadoop 2.9.2. The
> feature was brought up by JIRA
> https://issues.apache.org/jira/browse/HADOOP-15707 which is released in
> Hadoop 3.2+ (see "Fixed Version")
>
>
>
> Also, you may want to go through the Hadoop manual/doc for your current
> Hadoop version. If you change r3.3.0 to r2.9.2 in the URL you posted, you
> should be reading:
> https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
>
>
>
> Thanks,
>
>
>
> On Fri, Aug 21, 2020 at 11:02 AM Muthupandiyan, Kamaraj <
> kamaraj.muthupandiyan@hpe.com> wrote:
>
> Hello team,
>
>
>
> I need to configure LB for our name node setup and we are using Hadoop
> 2.9.2 . I was going through the docs from
> https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
> and found the below option, but whenever I am using the HTTP endpoints as
> mentioned below I am getting 404 response
>
>
>
>
>
> If you are running a set of NameNodes behind a Load Balancer (e.g. Azure
> <https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview>
> or AWS
> <https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> )
> and would like the Load Balancer to point to the active NN, you can use the
> /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive
> <http://nn_hostname/isActive> will return a 200 status code response if
> the NN is in Active HA State, 405 otherwise.
>
>
>
> Regards,
>
> Kamaraj
>
>
>
>
>
>
> --
>
> L
>
>
>
>
>
>
>
--
L
RE: setting up Load balancer for name nodes
Posted by "Muthupandiyan, Kamaraj" <ka...@hpe.com>.
Hi Team/Mingliang
We have upgraded to Hadoop 3.2.1 and we are having following issues , can somebody please help me on this.
We have successfully started name node and journal-node. However, while starting the Datanode from namenode is not working , the command simply exits.
[hdfs@sym-hdfsnn2 hadoop]$ hdfs version
Hadoop 3.2.1
Source code repository https://gitbox.apache.org/repos/asf/hadoop.git -r b3cbbb467e22ea829b3808f4b7b01d07e0bf3842
Compiled by rohithsharmaks on 2019-09-10T15:56Z
Compiled with protoc 2.5.0
From source with checksum 776eaf9eee9c0ffc370bcbc1888737
This command was run using /home/hdfs/hadoop/share/hadoop/common/hadoop-common-3.2.1.jar
[hdfs@sym-hdfsnn2 hadoop]$ hdfs --daemon stop datanode
Regards,
Kamaraj
From: Muthupandiyan, Kamaraj<ma...@hpe.com>
Sent: Sunday, August 23, 2020 9:22 PM
To: Mingliang Liu<ma...@gmail.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: setting up Load balancer for name nodes
Hi Mingliang Liu,
Could you please help me with my additional questions
1. Do we have any other options for setting up the Load balancer for name node in Hadoop Version 2.9.2 ?
1. I got the following document for upgrading the HDFS cluster - https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html or Do we have any other document for upgrading Version 2.9.2 to Version 3.2.x
Regards,
Kamaraj
InfoSight - SRE
From: Muthupandiyan, Kamaraj<ma...@hpe.com>
Sent: Sunday, August 23, 2020 6:57 PM
Subject: RE: setting up Load balancer for name nodes
Hi Mingliang Liu,
Thanks a lot for your response, I have read the document for Hadoop 3.3.0 and found that Load balancers will be supported on AWS/Azure, will it supported on A10 Loadbalancers or nginx ?
Regards,
Kamaraj
InfoSight - SRE
From: Mingliang Liu<ma...@gmail.com>
Sent: Saturday, August 22, 2020 2:55 AM
To: Muthupandiyan, Kamaraj<ma...@hpe.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: setting up Load balancer for name nodes
Hi Kamaraj,
I don't think the feature you are referring to exists in Hadoop 2.9.2. The feature was brought up by JIRA https://issues.apache.org/jira/browse/HADOOP-15707<https://issues.apache.org/jira/browse/HADOOP-15707> which is released in Hadoop 3.2+ (see "Fixed Version")
Also, you may want to go through the Hadoop manual/doc for your current Hadoop version. If you change r3.3.0 to r2.9.2 in the URL you posted, you should be reading: https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html>
Thanks,
On Fri, Aug 21, 2020 at 11:02 AM Muthupandiyan, Kamaraj <ka...@hpe.com>> wrote:
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html> and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
--
L
RE: setting up Load balancer for name nodes
Posted by "Muthupandiyan, Kamaraj" <ka...@hpe.com>.
Hi Mingliang Liu,
Could you please help me with my additional questions
1. Do we have any other options for setting up the Load balancer for name node in Hadoop Version 2.9.2 ?
1. I got the following document for upgrading the HDFS cluster - https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html or Do we have any other document for upgrading Version 2.9.2 to Version 3.2.x
Regards,
Kamaraj
InfoSight - SRE
From: Muthupandiyan, Kamaraj<ma...@hpe.com>
Sent: Sunday, August 23, 2020 6:57 PM
Subject: RE: setting up Load balancer for name nodes
Hi Mingliang Liu,
Thanks a lot for your response, I have read the document for Hadoop 3.3.0 and found that Load balancers will be supported on AWS/Azure, will it supported on A10 Loadbalancers or nginx ?
Regards,
Kamaraj
InfoSight - SRE
From: Mingliang Liu<ma...@gmail.com>
Sent: Saturday, August 22, 2020 2:55 AM
To: Muthupandiyan, Kamaraj<ma...@hpe.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: setting up Load balancer for name nodes
Hi Kamaraj,
I don't think the feature you are referring to exists in Hadoop 2.9.2. The feature was brought up by JIRA https://issues.apache.org/jira/browse/HADOOP-15707<https://issues.apache.org/jira/browse/HADOOP-15707> which is released in Hadoop 3.2+ (see "Fixed Version")
Also, you may want to go through the Hadoop manual/doc for your current Hadoop version. If you change r3.3.0 to r2.9.2 in the URL you posted, you should be reading: https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html>
Thanks,
On Fri, Aug 21, 2020 at 11:02 AM Muthupandiyan, Kamaraj <ka...@hpe.com>> wrote:
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html> and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
--
L
RE: setting up Load balancer for name nodes
Posted by "Muthupandiyan, Kamaraj" <ka...@hpe.com>.
Hi Mingliang Liu,
Thanks a lot for your response, I have read the document for Hadoop 3.3.0 and found that Load balancers will be supported on AWS/Azure, will it supported on A10 Loadbalancers or nginx ?
Regards,
Kamaraj
InfoSight - SRE
From: Mingliang Liu<ma...@gmail.com>
Sent: Saturday, August 22, 2020 2:55 AM
To: Muthupandiyan, Kamaraj<ma...@hpe.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: setting up Load balancer for name nodes
Hi Kamaraj,
I don't think the feature you are referring to exists in Hadoop 2.9.2. The feature was brought up by JIRA https://issues.apache.org/jira/browse/HADOOP-15707<https://issues.apache.org/jira/browse/HADOOP-15707> which is released in Hadoop 3.2+ (see "Fixed Version")
Also, you may want to go through the Hadoop manual/doc for your current Hadoop version. If you change r3.3.0 to r2.9.2 in the URL you posted, you should be reading: https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html>
Thanks,
On Fri, Aug 21, 2020 at 11:02 AM Muthupandiyan, Kamaraj <ka...@hpe.com>> wrote:
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html> and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
--
L
Re: setting up Load balancer for name nodes
Posted by Mingliang Liu <li...@gmail.com>.
Hi Kamaraj,
I don't think the feature you are referring to exists in Hadoop 2.9.2. The
feature was brought up by JIRA
https://issues.apache.org/jira/browse/HADOOP-15707 which is released in
Hadoop 3.2+ (see "Fixed Version")
Also, you may want to go through the Hadoop manual/doc for your current
Hadoop version. If you change r3.3.0 to r2.9.2 in the URL you posted, you
should be reading:
https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
Thanks,
On Fri, Aug 21, 2020 at 11:02 AM Muthupandiyan, Kamaraj <
kamaraj.muthupandiyan@hpe.com> wrote:
> Hello team,
>
>
>
> I need to configure LB for our name node setup and we are using Hadoop
> 2.9.2 . I was going through the docs from
> https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
> and found the below option, but whenever I am using the HTTP endpoints as
> mentioned below I am getting 404 response
>
>
>
>
>
> If you are running a set of NameNodes behind a Load Balancer (e.g. Azure
> <https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview>
> or AWS
> <https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> )
> and would like the Load Balancer to point to the active NN, you can use the
> /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive
> <http://nn_hostname/isActive> will return a 200 status code response if
> the NN is in Active HA State, 405 otherwise.
>
>
>
> Regards,
>
> Kamaraj
>
>
>
--
L
RE: setting up Load balancer for name nodes
Posted by "Muthupandiyan, Kamaraj" <ka...@hpe.com>.
Thank you Aaron, will go through the document accordingly.
Regards,
Kamaraj
From: Aaron Grubb<ma...@kaden.ai>
Sent: Saturday, August 22, 2020 12:17 AM
To: Muthupandiyan, Kamaraj<ma...@hpe.com>; user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: setting up Load balancer for name nodes
It is important to always read documents for your version as documentation for other version will not specify which versions contain the feature being documented. https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html>
From: Muthupandiyan, Kamaraj <ka...@hpe.com>
Sent: Friday, August 21, 2020 10:05 AM
To: user@hadoop.apache.org
Subject: setting up Load balancer for name nodes
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html> and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
RE: setting up Load balancer for name nodes
Posted by Aaron Grubb <aa...@kaden.ai>.
It is important to always read documents for your version as documentation for other version will not specify which versions contain the feature being documented. https://hadoop.apache.org/docs/r2.9.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
From: Muthupandiyan, Kamaraj <ka...@hpe.com>
Sent: Friday, August 21, 2020 10:05 AM
To: user@hadoop.apache.org
Subject: setting up Load balancer for name nodes
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
RE: setting up Load balancer for name nodes
Posted by "Muthupandiyan, Kamaraj" <ka...@hpe.com>.
Thank you Sanjeev, let me check that option
Regards,
Kamaraj
InfoSight - SRE
From: संजीव (Sanjeev Tripurari)<ma...@gmail.com>
Sent: Monday, August 24, 2020 10:14 AM
To: Muthupandiyan, Kamaraj<ma...@hpe.com>
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: setting up Load balancer for name nodes
Hi
In LB see if you can have health check script, if yes, then write a script which responds OK for the node which is active namenode, you can query namenode JMX to figure out active one.
Hope it helps.
Regards
-Sanjeev
On Fri, 21 Aug 2020 at 23:32, Muthupandiyan, Kamaraj <ka...@hpe.com>> wrote:
Hello team,
I need to configure LB for our name node setup and we are using Hadoop 2.9.2 . I was going through the docs from https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html<https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html> and found the below option, but whenever I am using the HTTP endpoints as mentioned below I am getting 404 response
If you are running a set of NameNodes behind a Load Balancer (e.g. Azure<https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview> or AWS<https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> ) and would like the Load Balancer to point to the active NN, you can use the /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive<http://nn_hostname/isActive> will return a 200 status code response if the NN is in Active HA State, 405 otherwise.
Regards,
Kamaraj
Re: setting up Load balancer for name nodes
Posted by "संजीव (Sanjeev Tripurari)" <sa...@gmail.com>.
Hi
In LB see if you can have health check script, if yes, then write a script
which responds OK for the node which is active namenode, you can query
namenode JMX to figure out active one.
Hope it helps.
Regards
-Sanjeev
On Fri, 21 Aug 2020 at 23:32, Muthupandiyan, Kamaraj <
kamaraj.muthupandiyan@hpe.com> wrote:
> Hello team,
>
>
>
> I need to configure LB for our name node setup and we are using Hadoop
> 2.9.2 . I was going through the docs from
> https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
> and found the below option, but whenever I am using the HTTP endpoints as
> mentioned below I am getting 404 response
>
>
>
>
>
> If you are running a set of NameNodes behind a Load Balancer (e.g. Azure
> <https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview>
> or AWS
> <https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-healthchecks.html> )
> and would like the Load Balancer to point to the active NN, you can use the
> /isActive HTTP endpoint as a health probe. http://NN_HOSTNAME/isActive
> <http://nn_hostname/isActive> will return a 200 status code response if
> the NN is in Active HA State, 405 otherwise.
>
>
>
> Regards,
>
> Kamaraj
>
>
>