You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@knox.apache.org by "Ruland, Benjamin" <Be...@computacenter.com> on 2016/05/13 11:11:51 UTC

Full Support of Hadoop Web UIs? (for instance YARN)

Hey guys,

for me, one of the main goals for using Knox is centralizing all Hadoop Management UIs in one place (on the edge node). But: Looking at the current status of Knox, for example for the WebUI of YARN, it seems that the support for those UIs is currently only implemented partially.

For instance, jumping to the nodemanagers, Applications masters or even parts of the Resource Manager UI does not work (at least in my test environment using the service-definitions out of the latest mirror of Knox).

I saw that the guys from IBM have extended the rewrite-Script a bit in their BigInsights Hadoop Distribution:
http://www-01.ibm.com/support/docview.wss?uid=swg21980169

Are there plans to further develop the UI-Redirections (for instance for YARN) or incorporate the progress of IBM?

Best regards,
Benjamin

AW: Full Support of Hadoop Web UIs? (for instance YARN)

Posted by "Ruland, Benjamin" <Be...@computacenter.com>.

Hi Sumit and Larry,

thanks for your replies and the reference to KNOX-711. I was not aware of that issue before, but can see the difficulties.

An exemplary issue is a OUT-rewrite rule that does not seem to fire:

On page https://knox-host:8443/gateway/default/yarn/apps the link "/cluster/app/application_1463126935438_0007" is not substituted to https://knox-host:8443/gateway/default/yarn/cluster/app/application_1463126935438_0007.
In my understanding, the following rule should have rewritten the link:
  <rule dir="OUT" name="YARNUI/yarn/outbound/cluster" pattern="/cluster/{**}">
    <rewrite template="{$frontend[url]}/yarn/{**}"/>
  </rule>

Furthermore, on the nodes pages (https://knox-host:8443/gateway/default/yarn/nodes), the node HTTP addresses are not substituted.

However, a possible reason might be, that I'm using the predelivered Knox version of HDP (which is 0.6) together with the service descriptions of the current Knox Github master.

Do you got any other hints for resolving (or debugging) this and similar issues?

Best,
Benjamin


Von: Sumit Gupta [mailto:sumit.gupta@hortonworks.com]
Gesendet: Freitag, 13. Mai 2016 15:42
An: user@knox.apache.org
Betreff: Re: Full Support of Hadoop Web UIs? (for instance YARN)

Hi Benjamin,

Thanks for the pointer to the IBM page. I will take a look at the new rules that they have added but also Larry is right in suspecting KNOX-711 for some of your issues. I do hope that the IBM guys continue to contribute changes in open source as they were the original contributors of the JS/HTML rewrite functionality.

Sumit.


From: larry mccay <la...@gmail.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Friday, May 13, 2016 at 7:26 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: Full Support of Hadoop Web UIs? (for instance YARN)

Hi Benjamin -

KNOX-711 is going to be working addressing issues related to the fact that rewrite rules for a service definition are treated globally to the topology by default.
The reason this was originally designed this way was for REST API responses that returned URLs that needed to be rewritten by other services.
For instance, Oozie returns payloads with WebHDFS URLs, etc.

When we started with UI proxying, the global rewrite rules more often match each other and result in improperly rewritten URLs.
If you look at the page source and find that some URLs have the wrong service/UI names then that is the issue that you are likely looking at.
This is often on the CSS files.

The current workaround is to put these UIs in separate topology files. This will unfortunately make the URLs differ from UI to UI for the time being but will not have the rules collide.

We are in the process of scoping our 0.9.1 release with this issue as one of the driving usecases.

Please do let me know if that is the issue that you are seeing.

thanks,

--larry

On Fri, May 13, 2016 at 7:11 AM, Ruland, Benjamin <Be...@computacenter.com>> wrote:
Hey guys,

for me, one of the main goals for using Knox is centralizing all Hadoop Management UIs in one place (on the edge node). But: Looking at the current status of Knox, for example for the WebUI of YARN, it seems that the support for those UIs is currently only implemented partially.

For instance, jumping to the nodemanagers, Applications masters or even parts of the Resource Manager UI does not work (at least in my test environment using the service-definitions out of the latest mirror of Knox).

I saw that the guys from IBM have extended the rewrite-Script a bit in their BigInsights Hadoop Distribution:
http://www-01.ibm.com/support/docview.wss?uid=swg21980169

Are there plans to further develop the UI-Redirections (for instance for YARN) or incorporate the progress of IBM?

Best regards,
Benjamin



-----------------------------------
Computacenter AG & Co. oHG, mit Sitz in Kerpen
(Amtsgericht Köln HRA 18096)
Vertretungsberechtigte Gesellschafter:
Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
Vorstand: Tony Conophy
Aufsichtsrat: Michael Norris (Vorsitzender)
Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
Geschäftsführer: Dr. Karsten Freihube, Dr. Christine Haupt, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
Visit us on the Internet: http://www.computacenter.de<http://www.computacenter.de/>
Visit our Online-Shop: https://shop.computacenter.de<https://shop.computacenter.de/>

This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
-----------------------------------

Re: Full Support of Hadoop Web UIs? (for instance YARN)

Posted by Sumit Gupta <su...@hortonworks.com>.

Hi Benjamin,

Thanks for the pointer to the IBM page. I will take a look at the new rules that they have added but also Larry is right in suspecting KNOX-711 for some of your issues. I do hope that the IBM guys continue to contribute changes in open source as they were the original contributors of the JS/HTML rewrite functionality.

Sumit.

From: larry mccay <la...@gmail.com>>
Reply-To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Date: Friday, May 13, 2016 at 7:26 AM
To: "user@knox.apache.org<ma...@knox.apache.org>" <us...@knox.apache.org>>
Subject: Re: Full Support of Hadoop Web UIs? (for instance YARN)

Hi Benjamin -

KNOX-711 is going to be working addressing issues related to the fact that rewrite rules for a service definition are treated globally to the topology by default.
The reason this was originally designed this way was for REST API responses that returned URLs that needed to be rewritten by other services.
For instance, Oozie returns payloads with WebHDFS URLs, etc.

When we started with UI proxying, the global rewrite rules more often match each other and result in improperly rewritten URLs.
If you look at the page source and find that some URLs have the wrong service/UI names then that is the issue that you are likely looking at.
This is often on the CSS files.

The current workaround is to put these UIs in separate topology files. This will unfortunately make the URLs differ from UI to UI for the time being but will not have the rules collide.

We are in the process of scoping our 0.9.1 release with this issue as one of the driving usecases.

Please do let me know if that is the issue that you are seeing.

thanks,

--larry

On Fri, May 13, 2016 at 7:11 AM, Ruland, Benjamin <Be...@computacenter.com>> wrote:
Hey guys,

for me, one of the main goals for using Knox is centralizing all Hadoop Management UIs in one place (on the edge node). But: Looking at the current status of Knox, for example for the WebUI of YARN, it seems that the support for those UIs is currently only implemented partially.

For instance, jumping to the nodemanagers, Applications masters or even parts of the Resource Manager UI does not work (at least in my test environment using the service-definitions out of the latest mirror of Knox).

I saw that the guys from IBM have extended the rewrite-Script a bit in their BigInsights Hadoop Distribution:
http://www-01.ibm.com/support/docview.wss?uid=swg21980169

Are there plans to further develop the UI-Redirections (for instance for YARN) or incorporate the progress of IBM?

Best regards,
Benjamin

-----------------------------------
Computacenter AG & Co. oHG, mit Sitz in Kerpen
(Amtsgericht Köln HRA 18096)
Vertretungsberechtigte Gesellschafter:
Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
Vorstand: Tony Conophy
Aufsichtsrat: Michael Norris (Vorsitzender)
Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
Geschäftsführer: Dr. Karsten Freihube, Dr. Christine Haupt, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
Visit us on the Internet: http://www.computacenter.de<http://www.computacenter.de/>
Visit our Online-Shop: https://shop.computacenter.de<https://shop.computacenter.de/>

This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
-----------------------------------

Re: Full Support of Hadoop Web UIs? (for instance YARN)

Posted by larry mccay <la...@gmail.com>.

Hi Benjamin -

KNOX-711 is going to be working addressing issues related to the fact that
rewrite rules for a service definition are treated globally to the topology
by default.
The reason this was originally designed this way was for REST API responses
that returned URLs that needed to be rewritten by other services.
For instance, Oozie returns payloads with WebHDFS URLs, etc.

When we started with UI proxying, the global rewrite rules more often match
each other and result in improperly rewritten URLs.
If you look at the page source and find that some URLs have the wrong
service/UI names then that is the issue that you are likely looking at.
This is often on the CSS files.

The current workaround is to put these UIs in separate topology files. This
will unfortunately make the URLs differ from UI to UI for the time being
but will not have the rules collide.

We are in the process of scoping our 0.9.1 release with this issue as one
of the driving usecases.

Please do let me know if that is the issue that you are seeing.

thanks,

--larry

On Fri, May 13, 2016 at 7:11 AM, Ruland, Benjamin <
Benjamin.Ruland@computacenter.com> wrote:

> Hey guys,
>
>
>
> for me, one of the main goals for using Knox is centralizing all Hadoop
> Management UIs in one place (on the edge node). But: Looking at the current
> status of Knox, for example for the WebUI of YARN, it seems that the
> support for those UIs is currently only implemented partially.
>
>
>
> For instance, jumping to the nodemanagers, Applications masters or even
> parts of the Resource Manager UI does not work (at least in my test
> environment using the service-definitions out of the latest mirror of Knox).
>
>
>
> I saw that the guys from IBM have extended the rewrite-Script a bit in
> their BigInsights Hadoop Distribution:
>
> http://www-01.ibm.com/support/docview.wss?uid=swg21980169
>
>
>
> Are there plans to further develop the UI-Redirections (for instance for
> YARN) or incorporate the progress of IBM?
>
>
>
> Best regards,
>
> Benjamin
>
>
>
>
> -----------------------------------
> Computacenter AG & Co. oHG, mit Sitz in Kerpen
> (Amtsgericht Köln HRA 18096)
> Vertretungsberechtigte Gesellschafter:
> Computacenter Aktiengesellschaft, mit Sitz in Köln (Amtsgericht Köln HRB 28384)
> Vorstand: Tony Conophy
> Aufsichtsrat: Michael Norris (Vorsitzender)
> Computacenter Management GmbH, mit Sitz in Köln (Amtsgericht Köln HRB 28284)
> Geschäftsführer: Dr. Karsten Freihube, Dr. Christine Haupt, Dr. Thomas Kottmann, Reiner Louis, Thomas Jescheck
> Visit us on the Internet: http://www.computacenter.de
> Visit our Online-Shop: https://shop.computacenter.de
>
> This email is confidential. If you are not the intended recipient, you must not disclose or use the information contained in it. If you have received this mail in error, please tell us immediately by return email and delete the document.
> -----------------------------------
>
>