You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by "Cecile, Adam" <Ad...@hitec.lu> on 2016/05/04 08:34:14 UTC

NameNode HA from a client perspective

Hello All,


I'd like to have a piece of advice regarding how my HDFS clients should handle the NameNode high availability feature.
I have a complete setup running with ZKFC and I can see one active and one standby NameNode. When I kill the active one, the standy gets active and when the original one get back online it turns into a standby node, perfect.

However I'm not sure how my client apps should handle this, a couple of ideas:
* Handle the bad HTTP code from standby node to switch to the other one
* Integrate Zookeeper client to query for the current active node
* Hack something like a shared-ip linked to the active node

Then I'll have to handle a switch that may occurs during the execution of a client app: should I just crash and rely on the cluster to restart the job.


Thanks in advance,

Best regards from Luxembourg.?

RE: NameNode HA from a client perspective

Posted by "Cecile, Adam" <Ad...@hitec.lu>.
?Sandeep, Bramha,


Thanks for you answer, that helps a lot! We'll see if we go for the custom StandByException handling or with the 3rd party module.


Regards, Adam.

________________________________
De : Sandeep Nemuri <nh...@gmail.com>
Envoy? : mercredi 4 mai 2016 11:18
? : Brahma Reddy Battula
Cc : Cecile, Adam; user@hadoop.apache.org
Objet : Re: NameNode HA from a client perspective

This could help you: https://pypi.python.org/pypi/PyHDFS/0.1.0

Thanks,
Sandeep
[https://mailfoogae.appspot.com/t?sender=abmhzYW5kZWVwNkBnbWFpbC5jb20%3D&type=zerocontent&guid=7e3cecdd-6d1b-4dbd-ba95-140c5f11e3a2]?

On Wed, May 4, 2016 at 2:40 PM, Brahma Reddy Battula <br...@huawei.com>> wrote:
1. Have a list of namenodes, built from configurations.
2. Execute the op on each namenode until its success.
3. Have the successfull namenode url as active namenode, and use the same for next operations.
4. Whenever a StandByException or some network exception (other than remote exceptions) occurs, then repeat #2 and #3, starting from the next namenode url in the list.


--Brahma Reddy Battula

From: Cecile, Adam [mailto:Adam.Cecile@hitec.lu<ma...@hitec.lu>]
Sent: 04 May 2016 16:26
To: Sandeep Nemuri
Cc: user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: RE: NameNode HA from a client perspective


Hello,



I'm not sure to understand your answer, may I add a little piece of code:



def _build_hdfs_url(self, hdfs_path, hdfs_operation, opt_query_param_tuples=[]):

        """

        :type hdfs_path: str

        :type hdfs_operation: str

        """

        if not hdfs_path.startswith("/"):

            raise WebHdfsException("The web hdfs path must start with / but found " + hdfs_path, None, None)



        url = 'http://' + self.host + ':' + str(self.port) + '/webhdfs/v1' + hdfs_path + '?user.name<http://user.name>=' + self.user + '&op=' + hdfs_operation

        len_param = len(opt_query_param_tuples)

        for index in range(len_param):

            key_value = opt_query_param_tuples[index]

            url += "&{}={}".format(key_value[0], str(key_value[1]))

        return url



Here is a plain python standard distribution function extracted from an app: the problem here is "self.host", it has to be IP address ou DNS name of the NameNode, however I'd like to turn this into something dynamic resolving to the current active master.



Regards, Adam.



________________________________
De : Sandeep Nemuri <nh...@gmail.com>>
Envoy? : mercredi 4 mai 2016 09:15
? : Cecile, Adam
Cc : user@hadoop.apache.org<ma...@hadoop.apache.org>
Objet : Re: NameNode HA from a client perspective

I think you can simply use the nameservice (dfs.nameservices) which is defined in hdfs-site.xml
The hdfs client should be able to resolve the current active namenode and get the necessary information.

Thanks,
Sandeep Nemuri
[https://mailfoogae.appspot.com/t?sender=abmhzYW5kZWVwNkBnbWFpbC5jb20%3D&type=zerocontent&guid=69e2c096-0009-4482-a881-df6dfb44434f]?

On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam <Ad...@hitec.lu>> wrote:

Hello All,


I'd like to have a piece of advice regarding how my HDFS clients should handle the NameNode high availability feature.
I have a complete setup running with ZKFC and I can see one active and one standby NameNode. When I kill the active one, the standy gets active and when the original one get back online it turns into a standby node, perfect.

However I'm not sure how my client apps should handle this, a couple of ideas:
* Handle the bad HTTP code from standby node to switch to the other one
* Integrate Zookeeper client to query for the current active node
* Hack something like a shared-ip linked to the active node

Then I'll have to handle a switch that may occurs during the execution of a client app: should I just crash and rely on the cluster to restart the job.


Thanks in advance,

Best regards from Luxembourg.?



--
  Regards
  Sandeep Nemuri



--
  Regards
  Sandeep Nemuri

Re: NameNode HA from a client perspective

Posted by Sandeep Nemuri <nh...@gmail.com>.
This could help you: https://pypi.python.org/pypi/PyHDFS/0.1.0

Thanks,
Sandeep
ᐧ

On Wed, May 4, 2016 at 2:40 PM, Brahma Reddy Battula <
brahmareddy.battula@huawei.com> wrote:

> 1. Have a list of namenodes, built from configurations.
> 2. Execute the op on each namenode until its success.
> 3. Have the successfull namenode url as active namenode, and use the same
> for next operations.
> 4. Whenever a StandByException or some network exception (other than
> remote exceptions) occurs, then repeat #2 and #3, starting from the next
> namenode url in the list.
>
>
>
>
>
> --Brahma Reddy Battula
>
>
>
> *From:* Cecile, Adam [mailto:Adam.Cecile@hitec.lu]
> *Sent:* 04 May 2016 16:26
> *To:* Sandeep Nemuri
> *Cc:* user@hadoop.apache.org
> *Subject:* RE: NameNode HA from a client perspective
>
>
>
> Hello,
>
>
>
> I'm not sure to understand your answer, may I add a little piece of code:
>
>
>
> *def* *_build_hdfs_url*(self, hdfs_path, hdfs_operation, opt_query_param_tuples*=*[]):
>
>         """
>
>         :type hdfs_path: str
>
>         :type hdfs_operation: str
>
>         """
>
>         *if* *not* hdfs_path*.*startswith("/"):
>
>             *raise* WebHdfsException("The web hdfs path must start with / but found " *+* hdfs_path, None, None)
>
>
>
>         url *=* 'http://' *+* self*.*host *+* ':' *+* str(self*.*port) *+* '/webhdfs/v1' *+* hdfs_path *+* '?user.name=' *+* self*.*user *+* '&op=' *+* hdfs_operation
>
>         len_param *=* len(opt_query_param_tuples)
>
>         *for* index *in* range(len_param):
>
>             key_value *=* opt_query_param_tuples[index]
>
>             url *+=* "&{}={}"*.*format(key_value[0], str(key_value[1]))
>
>         *return* url
>
>
>
> Here is a plain python standard distribution function extracted from an
> app: the problem here is "self.host", it has to be IP address ou DNS name
> of the NameNode, however I'd like to turn this into something dynamic
> resolving to the current active master.
>
>
>
> Regards, Adam.
>
>
> ------------------------------
>
> *De :* Sandeep Nemuri <nh...@gmail.com>
> *Envoyé :* mercredi 4 mai 2016 09:15
> *À :* Cecile, Adam
> *Cc :* user@hadoop.apache.org
> *Objet :* Re: NameNode HA from a client perspective
>
>
>
> I think you can simply use the nameservice (dfs.nameservices) which is
> defined in hdfs-site.xml
>
> The hdfs client should be able to resolve the current active namenode and
> get the necessary information.
>
>
>
> Thanks,
>
> Sandeep Nemuri
>
> ᐧ
>
>
>
> On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam <Ad...@hitec.lu>
> wrote:
>
> Hello All,
>
>
> I'd like to have a piece of advice regarding how my HDFS clients should
> handle the NameNode high availability feature.
> I have a complete setup running with ZKFC and I can see one active and one
> standby NameNode. When I kill the active one, the standy gets active and
> when the original one get back online it turns into a standby node, perfect.
>
> However I'm not sure how my client apps should handle this, a couple of
> ideas:
> * Handle the bad HTTP code from standby node to switch to the other one
> * Integrate Zookeeper client to query for the current active node
> * Hack something like a shared-ip linked to the active node
>
> Then I'll have to handle a switch that may occurs during the execution of
> a client app: should I just crash and rely on the cluster to restart the
> job.
>
>
> Thanks in advance,
>
> Best regards from Luxembourg.​
>
>
>
>
>
> --
>
> *  Regards*
>
> *  Sandeep Nemuri*
>



-- 
*  Regards*
*  Sandeep Nemuri*

RE: NameNode HA from a client perspective

Posted by Brahma Reddy Battula <br...@huawei.com>.
1. Have a list of namenodes, built from configurations.
2. Execute the op on each namenode until its success.
3. Have the successfull namenode url as active namenode, and use the same for next operations.
4. Whenever a StandByException or some network exception (other than remote exceptions) occurs, then repeat #2 and #3, starting from the next namenode url in the list.


--Brahma Reddy Battula

From: Cecile, Adam [mailto:Adam.Cecile@hitec.lu]
Sent: 04 May 2016 16:26
To: Sandeep Nemuri
Cc: user@hadoop.apache.org
Subject: RE: NameNode HA from a client perspective


Hello,



I'm not sure to understand your answer, may I add a little piece of code:



def _build_hdfs_url(self, hdfs_path, hdfs_operation, opt_query_param_tuples=[]):

        """

        :type hdfs_path: str

        :type hdfs_operation: str

        """

        if not hdfs_path.startswith("/"):

            raise WebHdfsException("The web hdfs path must start with / but found " + hdfs_path, None, None)



        url = 'http://' + self.host + ':' + str(self.port) + '/webhdfs/v1' + hdfs_path + '?user.name=' + self.user + '&op=' + hdfs_operation

        len_param = len(opt_query_param_tuples)

        for index in range(len_param):

            key_value = opt_query_param_tuples[index]

            url += "&{}={}".format(key_value[0], str(key_value[1]))

        return url



Here is a plain python standard distribution function extracted from an app: the problem here is "self.host", it has to be IP address ou DNS name of the NameNode, however I'd like to turn this into something dynamic resolving to the current active master.



Regards, Adam.



________________________________
De : Sandeep Nemuri <nh...@gmail.com>>
Envoyé : mercredi 4 mai 2016 09:15
À : Cecile, Adam
Cc : user@hadoop.apache.org<ma...@hadoop.apache.org>
Objet : Re: NameNode HA from a client perspective

I think you can simply use the nameservice (dfs.nameservices) which is defined in hdfs-site.xml
The hdfs client should be able to resolve the current active namenode and get the necessary information.

Thanks,
Sandeep Nemuri
[https://mailfoogae.appspot.com/t?sender=abmhzYW5kZWVwNkBnbWFpbC5jb20%3D&type=zerocontent&guid=69e2c096-0009-4482-a881-df6dfb44434f]ᐧ

On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam <Ad...@hitec.lu>> wrote:

Hello All,


I'd like to have a piece of advice regarding how my HDFS clients should handle the NameNode high availability feature.
I have a complete setup running with ZKFC and I can see one active and one standby NameNode. When I kill the active one, the standy gets active and when the original one get back online it turns into a standby node, perfect.

However I'm not sure how my client apps should handle this, a couple of ideas:
* Handle the bad HTTP code from standby node to switch to the other one
* Integrate Zookeeper client to query for the current active node
* Hack something like a shared-ip linked to the active node

Then I'll have to handle a switch that may occurs during the execution of a client app: should I just crash and rely on the cluster to restart the job.


Thanks in advance,

Best regards from Luxembourg.​



--
  Regards
  Sandeep Nemuri

RE: NameNode HA from a client perspective

Posted by "Cecile, Adam" <Ad...@hitec.lu>.
Hello,


I'm not sure to understand your answer, may I add a little piece of code:


def _build_hdfs_url(self, hdfs_path, hdfs_operation, opt_query_param_tuples=[]):
        """
        :type hdfs_path: str
        :type hdfs_operation: str
        """
        if not hdfs_path.startswith("/"):
            raise WebHdfsException("The web hdfs path must start with / but found " + hdfs_path, None, None)

        url = 'http://' + self.host + ':' + str(self.port) + '/webhdfs/v1' + hdfs_path + '?user.name=' + self.user + '&op=' + hdfs_operation
        len_param = len(opt_query_param_tuples)
        for index in range(len_param):
            key_value = opt_query_param_tuples[index]
            url += "&{}={}".format(key_value[0], str(key_value[1]))
        return url


Here is a plain python standard distribution function extracted from an app: the problem here is "self.host", it has to be IP address ou DNS name of the NameNode, however I'd like to turn this into something dynamic resolving to the current active master.


Regards, Adam.


________________________________
De : Sandeep Nemuri <nh...@gmail.com>
Envoy? : mercredi 4 mai 2016 09:15
? : Cecile, Adam
Cc : user@hadoop.apache.org
Objet : Re: NameNode HA from a client perspective

I think you can simply use the nameservice (dfs.nameservices) which is defined in hdfs-site.xml
The hdfs client should be able to resolve the current active namenode and get the necessary information.

Thanks,
Sandeep Nemuri
[https://mailfoogae.appspot.com/t?sender=abmhzYW5kZWVwNkBnbWFpbC5jb20%3D&type=zerocontent&guid=69e2c096-0009-4482-a881-df6dfb44434f]?

On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam <Ad...@hitec.lu>> wrote:

Hello All,


I'd like to have a piece of advice regarding how my HDFS clients should handle the NameNode high availability feature.
I have a complete setup running with ZKFC and I can see one active and one standby NameNode. When I kill the active one, the standy gets active and when the original one get back online it turns into a standby node, perfect.

However I'm not sure how my client apps should handle this, a couple of ideas:
* Handle the bad HTTP code from standby node to switch to the other one
* Integrate Zookeeper client to query for the current active node
* Hack something like a shared-ip linked to the active node

Then I'll have to handle a switch that may occurs during the execution of a client app: should I just crash and rely on the cluster to restart the job.


Thanks in advance,

Best regards from Luxembourg.?



--
  Regards
  Sandeep Nemuri

Re: NameNode HA from a client perspective

Posted by Sandeep Nemuri <nh...@gmail.com>.
I think you can simply use the nameservice (dfs.nameservices) which is
defined in hdfs-site.xml
The hdfs client should be able to resolve the current active namenode and
get the necessary information.

Thanks,
Sandeep Nemuri
ᐧ

On Wed, May 4, 2016 at 12:04 PM, Cecile, Adam <Ad...@hitec.lu> wrote:

> Hello All,
>
>
> I'd like to have a piece of advice regarding how my HDFS clients should
> handle the NameNode high availability feature.
> I have a complete setup running with ZKFC and I can see one active and one
> standby NameNode. When I kill the active one, the standy gets active and
> when the original one get back online it turns into a standby node, perfect.
>
> However I'm not sure how my client apps should handle this, a couple of
> ideas:
> * Handle the bad HTTP code from standby node to switch to the other one
> * Integrate Zookeeper client to query for the current active node
> * Hack something like a shared-ip linked to the active node
>
> Then I'll have to handle a switch that may occurs during the execution of
> a client app: should I just crash and rely on the cluster to restart the
> job.
>
>
> Thanks in advance,
>
> Best regards from Luxembourg.​
>



-- 
*  Regards*
*  Sandeep Nemuri*