You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by Mohammad Islam <mi...@yahoo.com> on 2016/11/01 19:14:06 UTC
HA support for YARN and NameNode UI level
Hi,I'm wandering if there is any built-in support in Knox for any (NN/RM) UI to automatically forward to active NN/RM.
I saw there is HA support in WebHDFS level. But not sure if I need to do anything special for HDFSUI. The same is true for YARNUI.
Regards,Mohammad
Re: HA support for YARN and NameNode UI level
Posted by larry mccay <lm...@apache.org>.
+1 on the collaborative approach within JIRA.
It's a great way to keep things moving while not stepping on each other's
toes and completely within the spirit of ASF!
On Thu, Nov 3, 2016 at 10:02 AM, Sumit Gupta <su...@hortonworks.com>
wrote:
> Hey Mohammad,
>
> Thanks for your interest and gumption in moving Knox forward. We always
> encourage collaboration and contributions of all kinds. I would be
> surprised if Jeffrey objected to any work or help with a patch. I¹m sure
> he would be in fact happy to review and/or work with you on a patch. I
> would recommend tagging Jeffrey in a Jira comment and going forward with
> working on a contribution. I say this also because the Jira has been
> created a while ago and could certainly use some progress.
>
> Thanks again!
> Sumit
>
>
> On 11/2/16, 8:27 PM, "Mohammad Islam" <mi...@yahoo.com.INVALID> wrote:
>
> >Thanks Sumit for the pointer.
> >Is there any plan to work on it soon? The reason I'm asking is : if I get
> >some guidance, I can also stab on it if Jeffrey (current Assignee) is
> >busy.
> >Regards,Mohammad
> >
> >
> > On Wednesday, November 2, 2016 4:43 PM, Sumit Gupta
> ><su...@hortonworks.com> wrote:
> >
> >
> > Hi Mohammad,
> >
> >You are right we have HA support for various services that is somewhat
> >generic and based on connectivity failures. Our WebHDFS HA support is a
> >bit more specific and maps to some of the nuanced states that the
> >Namenodes can be in. We have yet to put in specific support for Yarn and
> >like you point out there is a redirect that occurs that needs to be
> >handled.
> >
> >There is an open JIRA to add support for Yarn¹s resource manager in HA
> >mode.
> >
> >https://issues.apache.org/jira/browse/KNOX-719
> >
> >Sumit
> >
> >From: Mohammad Islam <mi...@yahoo.com>>
> >Reply-To: "user@knox.apache.org<ma...@knox.apache.org>"
> ><us...@knox.apache.org>>, Mohammad Islam
> ><mi...@yahoo.com>>
> >Date: Wednesday, November 2, 2016 at 6:57 PM
> >To: "user@knox.apache.org<ma...@knox.apache.org>"
> ><us...@knox.apache.org>>,
> >"dev@knox.apache.org<ma...@knox.apache.org>"
> ><de...@knox.apache.org>>
> >Subject: Re: HA support for YARN and NameNode UI level
> >
> >Just to add more specific use-case of YARN UI.
> >
> >BTW my KNOX service was running behind another FE proxy.
> >
> >When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I
> >saw the message " This is standby RM. Redirecting to the current active
> >RM:
> >http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster<http://<
> PRIMARY_UI_HOST>
> >:8088/cluster>".
> >
> >At this time, my browser also tried to connect to
> >"http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster<http://<
> PRIMARY_UI_HOST
> >>:8088/cluster>" but it's timed out because the YARN internal host was
> >>behind the firewall and was not directly accessible through browser.
> >
> >
> >Looks like Knox was sending a response to FE proxy with a header
> >"Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My
> >question is : is there a way for Knox to replace the internal Yarn Host
> >or endpoint with something like knoxy endpoint.
> >
> >Regards,
> >Mohammad
> >
> >
> >
> >
> >On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam
> ><mi...@yahoo.com>> wrote:
> >
> >
> >Hi,
> >I'm wandering if there is any built-in support in Knox for any (NN/RM) UI
> >to automatically forward to active NN/RM.
> >
> >I saw there is HA support in WebHDFS level. But not sure if I need to do
> >anything special for HDFSUI.
> >The same is true for YARNUI.
> >
> >Regards,
> >Mohammad
> >
> >
> >
> >
> >
> >
>
>
Re: HA support for YARN and NameNode UI level
Posted by Sumit Gupta <su...@hortonworks.com>.
Hey Mohammad,
Thanks for your interest and gumption in moving Knox forward. We always
encourage collaboration and contributions of all kinds. I would be
surprised if Jeffrey objected to any work or help with a patch. I¹m sure
he would be in fact happy to review and/or work with you on a patch. I
would recommend tagging Jeffrey in a Jira comment and going forward with
working on a contribution. I say this also because the Jira has been
created a while ago and could certainly use some progress.
Thanks again!
Sumit
On 11/2/16, 8:27 PM, "Mohammad Islam" <mi...@yahoo.com.INVALID> wrote:
>Thanks Sumit for the pointer.
>Is there any plan to work on it soon? The reason I'm asking is : if I get
>some guidance, I can also stab on it if Jeffrey (current Assignee) is
>busy.
>Regards,Mohammad
>
>
> On Wednesday, November 2, 2016 4:43 PM, Sumit Gupta
><su...@hortonworks.com> wrote:
>
>
> Hi Mohammad,
>
>You are right we have HA support for various services that is somewhat
>generic and based on connectivity failures. Our WebHDFS HA support is a
>bit more specific and maps to some of the nuanced states that the
>Namenodes can be in. We have yet to put in specific support for Yarn and
>like you point out there is a redirect that occurs that needs to be
>handled.
>
>There is an open JIRA to add support for Yarn¹s resource manager in HA
>mode.
>
>https://issues.apache.org/jira/browse/KNOX-719
>
>Sumit
>
>From: Mohammad Islam <mi...@yahoo.com>>
>Reply-To: "user@knox.apache.org<ma...@knox.apache.org>"
><us...@knox.apache.org>>, Mohammad Islam
><mi...@yahoo.com>>
>Date: Wednesday, November 2, 2016 at 6:57 PM
>To: "user@knox.apache.org<ma...@knox.apache.org>"
><us...@knox.apache.org>>,
>"dev@knox.apache.org<ma...@knox.apache.org>"
><de...@knox.apache.org>>
>Subject: Re: HA support for YARN and NameNode UI level
>
>Just to add more specific use-case of YARN UI.
>
>BTW my KNOX service was running behind another FE proxy.
>
>When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I
>saw the message " This is standby RM. Redirecting to the current active
>RM:
>http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster<http://<PRIMARY_UI_HOST>
>:8088/cluster>".
>
>At this time, my browser also tried to connect to
>"http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster<http://<PRIMARY_UI_HOST
>>:8088/cluster>" but it's timed out because the YARN internal host was
>>behind the firewall and was not directly accessible through browser.
>
>
>Looks like Knox was sending a response to FE proxy with a header
>"Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My
>question is : is there a way for Knox to replace the internal Yarn Host
>or endpoint with something like knoxy endpoint.
>
>Regards,
>Mohammad
>
>
>
>
>On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam
><mi...@yahoo.com>> wrote:
>
>
>Hi,
>I'm wandering if there is any built-in support in Knox for any (NN/RM) UI
>to automatically forward to active NN/RM.
>
>I saw there is HA support in WebHDFS level. But not sure if I need to do
>anything special for HDFSUI.
>The same is true for YARNUI.
>
>Regards,
>Mohammad
>
>
>
>
>
>
Re: HA support for YARN and NameNode UI level
Posted by Sumit Gupta <su...@hortonworks.com>.
Hey Mohammad,
Thanks for your interest and gumption in moving Knox forward. We always
encourage collaboration and contributions of all kinds. I would be
surprised if Jeffrey objected to any work or help with a patch. I¹m sure
he would be in fact happy to review and/or work with you on a patch. I
would recommend tagging Jeffrey in a Jira comment and going forward with
working on a contribution. I say this also because the Jira has been
created a while ago and could certainly use some progress.
Thanks again!
Sumit
On 11/2/16, 8:27 PM, "Mohammad Islam" <mi...@yahoo.com.INVALID> wrote:
>Thanks Sumit for the pointer.
>Is there any plan to work on it soon? The reason I'm asking is : if I get
>some guidance, I can also stab on it if Jeffrey (current Assignee) is
>busy.
>Regards,Mohammad
>
>
> On Wednesday, November 2, 2016 4:43 PM, Sumit Gupta
><su...@hortonworks.com> wrote:
>
>
> Hi Mohammad,
>
>You are right we have HA support for various services that is somewhat
>generic and based on connectivity failures. Our WebHDFS HA support is a
>bit more specific and maps to some of the nuanced states that the
>Namenodes can be in. We have yet to put in specific support for Yarn and
>like you point out there is a redirect that occurs that needs to be
>handled.
>
>There is an open JIRA to add support for Yarn¹s resource manager in HA
>mode.
>
>https://issues.apache.org/jira/browse/KNOX-719
>
>Sumit
>
>From: Mohammad Islam <mi...@yahoo.com>>
>Reply-To: "user@knox.apache.org<ma...@knox.apache.org>"
><us...@knox.apache.org>>, Mohammad Islam
><mi...@yahoo.com>>
>Date: Wednesday, November 2, 2016 at 6:57 PM
>To: "user@knox.apache.org<ma...@knox.apache.org>"
><us...@knox.apache.org>>,
>"dev@knox.apache.org<ma...@knox.apache.org>"
><de...@knox.apache.org>>
>Subject: Re: HA support for YARN and NameNode UI level
>
>Just to add more specific use-case of YARN UI.
>
>BTW my KNOX service was running behind another FE proxy.
>
>When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I
>saw the message " This is standby RM. Redirecting to the current active
>RM:
>http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster<http://<PRIMARY_UI_HOST>
>:8088/cluster>".
>
>At this time, my browser also tried to connect to
>"http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster<http://<PRIMARY_UI_HOST
>>:8088/cluster>" but it's timed out because the YARN internal host was
>>behind the firewall and was not directly accessible through browser.
>
>
>Looks like Knox was sending a response to FE proxy with a header
>"Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My
>question is : is there a way for Knox to replace the internal Yarn Host
>or endpoint with something like knoxy endpoint.
>
>Regards,
>Mohammad
>
>
>
>
>On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam
><mi...@yahoo.com>> wrote:
>
>
>Hi,
>I'm wandering if there is any built-in support in Knox for any (NN/RM) UI
>to automatically forward to active NN/RM.
>
>I saw there is HA support in WebHDFS level. But not sure if I need to do
>anything special for HDFSUI.
>The same is true for YARNUI.
>
>Regards,
>Mohammad
>
>
>
>
>
>
Re: HA support for YARN and NameNode UI level
Posted by Mohammad Islam <mi...@yahoo.com>.
Thanks Sumit for the pointer.
Is there any plan to work on it soon? The reason I'm asking is : if I get some guidance, I can also stab on it if Jeffrey (current Assignee) is busy.
Regards,Mohammad
On Wednesday, November 2, 2016 4:43 PM, Sumit Gupta <su...@hortonworks.com> wrote:
Hi Mohammad,
You are right we have HA support for various services that is somewhat generic and based on connectivity failures. Our WebHDFS HA support is a bit more specific and maps to some of the nuanced states that the Namenodes can be in. We have yet to put in specific support for Yarn and like you point out there is a redirect that occurs that needs to be handled.
There is an open JIRA to add support for Yarn’s resource manager in HA mode.
https://issues.apache.org/jira/browse/KNOX-719
Sumit
From: Mohammad Islam <mi...@yahoo.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, Mohammad Islam <mi...@yahoo.com>>
Date: Wednesday, November 2, 2016 at 6:57 PM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, "dev@knox.apache.org<ma...@knox.apache.org>" <de...@knox.apache.org>>
Subject: Re: HA support for YARN and NameNode UI level
Just to add more specific use-case of YARN UI.
BTW my KNOX service was running behind another FE proxy.
When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I saw the message " This is standby RM. Redirecting to the current active RM: http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>".
At this time, my browser also tried to connect to "http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>" but it's timed out because the YARN internal host was behind the firewall and was not directly accessible through browser.
Looks like Knox was sending a response to FE proxy with a header "Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My question is : is there a way for Knox to replace the internal Yarn Host or endpoint with something like knoxy endpoint.
Regards,
Mohammad
On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam <mi...@yahoo.com>> wrote:
Hi,
I'm wandering if there is any built-in support in Knox for any (NN/RM) UI to automatically forward to active NN/RM.
I saw there is HA support in WebHDFS level. But not sure if I need to do anything special for HDFSUI.
The same is true for YARNUI.
Regards,
Mohammad
Re: HA support for YARN and NameNode UI level
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Thanks Sumit for the pointer.
Is there any plan to work on it soon? The reason I'm asking is : if I get some guidance, I can also stab on it if Jeffrey (current Assignee) is busy.
Regards,Mohammad
On Wednesday, November 2, 2016 4:43 PM, Sumit Gupta <su...@hortonworks.com> wrote:
Hi Mohammad,
You are right we have HA support for various services that is somewhat generic and based on connectivity failures. Our WebHDFS HA support is a bit more specific and maps to some of the nuanced states that the Namenodes can be in. We have yet to put in specific support for Yarn and like you point out there is a redirect that occurs that needs to be handled.
There is an open JIRA to add support for Yarn’s resource manager in HA mode.
https://issues.apache.org/jira/browse/KNOX-719
Sumit
From: Mohammad Islam <mi...@yahoo.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, Mohammad Islam <mi...@yahoo.com>>
Date: Wednesday, November 2, 2016 at 6:57 PM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, "dev@knox.apache.org<ma...@knox.apache.org>" <de...@knox.apache.org>>
Subject: Re: HA support for YARN and NameNode UI level
Just to add more specific use-case of YARN UI.
BTW my KNOX service was running behind another FE proxy.
When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I saw the message " This is standby RM. Redirecting to the current active RM: http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>".
At this time, my browser also tried to connect to "http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>" but it's timed out because the YARN internal host was behind the firewall and was not directly accessible through browser.
Looks like Knox was sending a response to FE proxy with a header "Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My question is : is there a way for Knox to replace the internal Yarn Host or endpoint with something like knoxy endpoint.
Regards,
Mohammad
On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam <mi...@yahoo.com>> wrote:
Hi,
I'm wandering if there is any built-in support in Knox for any (NN/RM) UI to automatically forward to active NN/RM.
I saw there is HA support in WebHDFS level. But not sure if I need to do anything special for HDFSUI.
The same is true for YARNUI.
Regards,
Mohammad
Re: HA support for YARN and NameNode UI level
Posted by Sumit Gupta <su...@hortonworks.com>.
Hi Mohammad,
You are right we have HA support for various services that is somewhat generic and based on connectivity failures. Our WebHDFS HA support is a bit more specific and maps to some of the nuanced states that the Namenodes can be in. We have yet to put in specific support for Yarn and like you point out there is a redirect that occurs that needs to be handled.
There is an open JIRA to add support for Yarn’s resource manager in HA mode.
https://issues.apache.org/jira/browse/KNOX-719
Sumit
From: Mohammad Islam <mi...@yahoo.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, Mohammad Islam <mi...@yahoo.com>>
Date: Wednesday, November 2, 2016 at 6:57 PM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, "dev@knox.apache.org<ma...@knox.apache.org>" <de...@knox.apache.org>>
Subject: Re: HA support for YARN and NameNode UI level
Just to add more specific use-case of YARN UI.
BTW my KNOX service was running behind another FE proxy.
When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I saw the message " This is standby RM. Redirecting to the current active RM: http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>".
At this time, my browser also tried to connect to "http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>" but it's timed out because the YARN internal host was behind the firewall and was not directly accessible through browser.
Looks like Knox was sending a response to FE proxy with a header "Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My question is : is there a way for Knox to replace the internal Yarn Host or endpoint with something like knoxy endpoint.
Regards,
Mohammad
On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam <mi...@yahoo.com>> wrote:
Hi,
I'm wandering if there is any built-in support in Knox for any (NN/RM) UI to automatically forward to active NN/RM.
I saw there is HA support in WebHDFS level. But not sure if I need to do anything special for HDFSUI.
The same is true for YARNUI.
Regards,
Mohammad
Re: HA support for YARN and NameNode UI level
Posted by Sumit Gupta <su...@hortonworks.com>.
Hi Mohammad,
You are right we have HA support for various services that is somewhat generic and based on connectivity failures. Our WebHDFS HA support is a bit more specific and maps to some of the nuanced states that the Namenodes can be in. We have yet to put in specific support for Yarn and like you point out there is a redirect that occurs that needs to be handled.
There is an open JIRA to add support for Yarn’s resource manager in HA mode.
https://issues.apache.org/jira/browse/KNOX-719
Sumit
From: Mohammad Islam <mi...@yahoo.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, Mohammad Islam <mi...@yahoo.com>>
Date: Wednesday, November 2, 2016 at 6:57 PM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>, "dev@knox.apache.org<ma...@knox.apache.org>" <de...@knox.apache.org>>
Subject: Re: HA support for YARN and NameNode UI level
Just to add more specific use-case of YARN UI.
BTW my KNOX service was running behind another FE proxy.
When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I saw the message " This is standby RM. Redirecting to the current active RM: http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>".
At this time, my browser also tried to connect to "http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster<http://<PRIMARY_UI_HOST>:8088/cluster>" but it's timed out because the YARN internal host was behind the firewall and was not directly accessible through browser.
Looks like Knox was sending a response to FE proxy with a header "Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My question is : is there a way for Knox to replace the internal Yarn Host or endpoint with something like knoxy endpoint.
Regards,
Mohammad
On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam <mi...@yahoo.com>> wrote:
Hi,
I'm wandering if there is any built-in support in Knox for any (NN/RM) UI to automatically forward to active NN/RM.
I saw there is HA support in WebHDFS level. But not sure if I need to do anything special for HDFSUI.
The same is true for YARNUI.
Regards,
Mohammad
Re: HA support for YARN and NameNode UI level
Posted by Mohammad Islam <mi...@yahoo.com.INVALID>.
Just to add more specific use-case of YARN UI.
BTW my KNOX service was running behind another FE proxy.
When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I saw the message " This is standby RM. Redirecting to the current active RM: http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster".
At this time, my browser also tried to connect to "http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster" but it's timed out because the YARN internal host was behind the firewall and was not directly accessible through browser.
Looks like Knox was sending a response to FE proxy with a header "Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My question is : is there a way for Knox to replace the internal Yarn Host or endpoint with something like knoxy endpoint.
Regards,Mohammad
On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam <mi...@yahoo.com> wrote:
Hi,I'm wandering if there is any built-in support in Knox for any (NN/RM) UI to automatically forward to active NN/RM.
I saw there is HA support in WebHDFS level. But not sure if I need to do anything special for HDFSUI. The same is true for YARNUI.
Regards,Mohammad
Re: HA support for YARN and NameNode UI level
Posted by Mohammad Islam <mi...@yahoo.com>.
Just to add more specific use-case of YARN UI.
BTW my KNOX service was running behind another FE proxy.
When I typed https://<MY_FE_PROXY>/gateway/sandbox/yarn in my browser, I saw the message " This is standby RM. Redirecting to the current active RM: http://<PRIMARY_DIRECT_YARN_UI_HOST>:8088/cluster".
At this time, my browser also tried to connect to "http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster" but it's timed out because the YARN internal host was behind the firewall and was not directly accessible through browser.
Looks like Knox was sending a response to FE proxy with a header "Refresh: 3; url=http://<DIRECT_YARN_UI_HOST_PRIMARY>:8088/cluster". My question is : is there a way for Knox to replace the internal Yarn Host or endpoint with something like knoxy endpoint.
Regards,Mohammad
On Tuesday, November 1, 2016 12:24 PM, Mohammad Islam <mi...@yahoo.com> wrote:
Hi,I'm wandering if there is any built-in support in Knox for any (NN/RM) UI to automatically forward to active NN/RM.
I saw there is HA support in WebHDFS level. But not sure if I need to do anything special for HDFSUI. The same is true for YARNUI.
Regards,Mohammad