You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by Praveen krishnamoorthy Ravikumar <pr...@njit.edu> on 2019/07/18 19:11:59 UTC

Does KNOX support redirection links

Greetings,

I’m working currently on enabling Amazon EMR debugging UIs running in private subnet via apache Knox. With the steps defined in the documentation I was able to install Knox and access YARN/SparkHistory/Ganglia UIs, which was amazing. But I’m facing issues on accessing certain links particularly the redirection links in NODE MANAGER UI.

My first question, Does the Knox supports redirection internally ?. For instance I was trying to access the container log message in the node manager UI (NODEMANAGERUI -> Local Logs [under tools] -> containers/ and was getting blocked with the browser URL - http://ip-10X.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042 <http://ip-10x.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042>


What I noticed was the links that returning Http 3xx are getting broken.

Could anyone please help me resolving this issue ?

Thanks,
Praveen.

Log message in gateway.log :
—————————————

2019-07-18 18:41:50,960 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /node/logs/containers

2019-07-18 18:41:50,963 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://cas-query-sandbox-.emr.us-east-1.opse.c1.com:443/gateway/cto/node/logs/containers?host=ip-10-X.us-east-1.opse.c1.com&port=8042 <https://cas-query-sandbox-sc-866316499622-pp-oeobt24ki6aja.emr.us-east-1.opse.c1.vanguard.com/gateway/cto/node/logs/containers?host=ip-10-156-232-245.us-east-1.opse.c1.vanguard.com&port=8042>, direction: IN via implicit rule: NODEUI/logs/containers to URL: http://ip-10-156X.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042 <http://ip-10-156x.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042>

2019-07-18 18:41:50,964 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://cas-query-sandbox.emr.us-east-1.opse.c1.com/gateway/cto/node/logs/?host=ip-X.us-east-1.opse.c1.com&port=8042 <https://cas-query-sandbox.emr.us-east-1.opse.c1.com/gateway/cto/node/logs/?host=ip-X.us-east-1.opse.c1.com&port=8042>, direction: IN via implicit rule: NODEUI/logs to URL: http://ip-10X.us-east-1.opse.c1.com:8042/logs/ <http://ip-10x.us-east-1.opse.c1.com:8042/logs/>

2019-07-18 18:41:50,964 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET http://ipX.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.com&port=8042 <http://ipx.us-east-1.opse.c1.vanguard.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.vanguard.com&port=8042>

2019-07-18 18:41:50,967 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 302

2019-07-18 18:41:50,968 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit character set UTF-8 for entity of type text/plain

2019-07-18 18:41:50,968 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound response entity content type: text/plain; charset=utf-8

Re: Does KNOX support redirection links

Posted by Sandeep Moré <mo...@gmail.com>.
Praveen,

Can you check the response http headers that you get back that i causing
this redirection, just want to make sure we are fighting the right problem.
You can see that in in browser developer console.
The yarnuiv2 example I was talking about modifies the Location header, this
is the example
https://github.com/apache/knox/blob/512147f6fd4677baa8f8ffed6034fac256157337/gateway-service-definitions/src/main/resources/services/yarnuiv2/3.0.0/rewrite.xml#L168

Best,
Sandeep

On Thu, Jul 18, 2019 at 4:55 PM Praveen krishnamoorthy Ravikumar <
pr334@njit.edu> wrote:

> Hello Sandeep,
>
> Thank you so much for your response.
>
> In the below case -> gateway.log
>
> 2019-07-18 20:34:31,279 DEBUG knox.gateway
> (GatewayFilter.java:doFilter(119)) - Received request: GET
> /node/logs/containers
>
> 2019-07-18 20:34:31,280 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://ec2-3-80-173-38.compute-1.amazonaws.com:8446/gateway/gate1/node/logs/containers?host=ip-172-31-8-223.ec2.internal&port=8042,
> direction: IN via implicit rule: NODEUI/logs2 to URL:
> http://ip-172-31-8-223.ec2.internal:8042/logs/containers
>
> 2019-07-18 20:34:31,281 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://ec2-3-80-173-38.compute-1.amazonaws.com:8446/gateway/gate1/node/logs/?host=ip-172-31-8-223.ec2.internal&port=8042,
> direction: IN via implicit rule: NODEUI/logs to URL:
> http://ip-172-31-8-223.ec2.internal:8042/logs/
>
> 2019-07-18 20:34:31,281 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
> http://ip-172-31-8-223.ec2.internal:8042/logs/containers?user.name=admin
>
> 2019-07-18 20:34:31,285 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
> status: 302
> 2019-07-18 20:34:31,285 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit
> character set UTF-8 for entity of type text/plain
> 2019-07-18 20:34:31,285 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound
> response entity content type: text/plain; charset=utf-8
>
> I see it is redirecting to “http://ip-172X:8042/logs/containers?user.admin
> <http://ip-172x:8042/logs/containers?user.admin>” - which is the direct
> ip link to the slave node.
>
> So Do I need to define seperate redirection rules to make it work?.
>
> Can you also share me yarnuiv2 service def links that I can refer, would
> be very helpful ?
>
>
> Thanks,
> Praveen.
>
> On Jul 18, 2019, at 4:36 PM, Sandeep Moré <mo...@gmail.com> wrote:
>
> Hello Praveen,
>
> If the UI is using Location header to redirect (which is most of the
> times) then Knox can do it, there are lot's of examples in the service defs
> where we do it (e.g. yarnuiv2)
>
> Best,
> Sandeep
>
> On Thu, Jul 18, 2019 at 3:12 PM Praveen krishnamoorthy Ravikumar <
> pr334@njit.edu> wrote:
>
>> Greetings,
>>
>> I’m working currently on enabling Amazon EMR debugging UIs running in
>> private subnet via apache Knox. With the steps defined in the documentation
>> I was able to install Knox and access YARN/SparkHistory/Ganglia UIs, which
>> was amazing. But I’m facing issues on accessing certain links particularly
>> the redirection links in NODE MANAGER UI.
>>
>> My first question, Does the Knox supports redirection internally ?. For
>> instance I was trying to access the container log message in the node
>> manager UI (NODEMANAGERUI -> Local Logs [under tools] -> containers/ and
>> was getting blocked with the browser URL -
>> http://ip-10X.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042
>> <http://ip-10x.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042>
>>
>>
>> What I noticed was the links that returning Http 3xx are getting broken.
>>
>> Could anyone please help me resolving this issue ?
>>
>> Thanks,
>> Praveen.
>>
>> Log message in gateway.log :
>> —————————————
>>
>> 2019-07-18 18:41:50,960 DEBUG knox.gateway
>> (GatewayFilter.java:doFilter(119)) - Received request: GET
>> /node/logs/containers
>>
>> 2019-07-18 18:41:50,963 DEBUG knox.gateway
>> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
>> https://cas-query-sandbox-.emr.us-east-1.opse.c1.com:443/gateway/cto/node/logs/containers?host=ip-10-X.us-east-1.opse.c1.com&port=8042
>> <https://cas-query-sandbox-sc-866316499622-pp-oeobt24ki6aja.emr.us-east-1.opse.c1.vanguard.com/gateway/cto/node/logs/containers?host=ip-10-156-232-245.us-east-1.opse.c1.vanguard.com&port=8042>,
>> direction: IN via implicit rule: NODEUI/logs/containers to URL:
>> http://ip-10-156X.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042
>> <http://ip-10-156x.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042>
>>
>> 2019-07-18 18:41:50,964 DEBUG knox.gateway
>> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
>> https://cas-query-sandbox.emr.us-east-1.opse.c1.com/gateway/cto/node/logs/?host=ip-X.us-east-1.opse.c1.com&port=8042,
>> direction: IN via implicit rule: NODEUI/logs to URL:
>> http://ip-10X.us-east-1.opse.c1.com:8042/logs/
>> <http://ip-10x.us-east-1.opse.c1.com:8042/logs/>
>>
>> 2019-07-18 18:41:50,964 DEBUG knox.gateway
>> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
>> http://ipX.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.com&port=8042
>> <http://ipx.us-east-1.opse.c1.vanguard.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.vanguard.com&port=8042>
>>
>> 2019-07-18 18:41:50,967 DEBUG knox.gateway
>> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
>> status: *302*
>>
>> 2019-07-18 18:41:50,968 DEBUG knox.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit
>> character set UTF-8 for entity of type text/plain
>>
>> 2019-07-18 18:41:50,968 DEBUG knox.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound
>> response entity content type: text/plain; charset=utf-8
>>
>>
>>
>

Re: Does KNOX support redirection links

Posted by Praveen krishnamoorthy Ravikumar <pr...@njit.edu>.
Hello Sandeep,

Thank you so much for your response.

In the below case -> gateway.log 

2019-07-18 20:34:31,279 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /node/logs/containers

2019-07-18 20:34:31,280 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://ec2-3-80-173-38.compute-1.amazonaws.com:8446/gateway/gate1/node/logs/containers?host=ip-172-31-8-223.ec2.internal&port=8042, direction: IN via implicit rule: NODEUI/logs2 to URL: http://ip-172-31-8-223.ec2.internal:8042/logs/containers

2019-07-18 20:34:31,281 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://ec2-3-80-173-38.compute-1.amazonaws.com:8446/gateway/gate1/node/logs/?host=ip-172-31-8-223.ec2.internal&port=8042, direction: IN via implicit rule: NODEUI/logs to URL: http://ip-172-31-8-223.ec2.internal:8042/logs/

2019-07-18 20:34:31,281 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET http://ip-172-31-8-223.ec2.internal:8042/logs/containers?user.name=admin

2019-07-18 20:34:31,285 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 302
2019-07-18 20:34:31,285 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit character set UTF-8 for entity of type text/plain
2019-07-18 20:34:31,285 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound response entity content type: text/plain; charset=utf-8

I see it is redirecting to “http://ip-172X:8042/logs/containers?user.admin” - which is the direct ip link to the slave node.

So Do I need to define seperate redirection rules to make it work?.

Can you also share me yarnuiv2 service def links that I can refer, would be very helpful ?


Thanks,
Praveen.

> On Jul 18, 2019, at 4:36 PM, Sandeep Moré <mo...@gmail.com> wrote:
> 
> Hello Praveen,
> 
> If the UI is using Location header to redirect (which is most of the times) then Knox can do it, there are lot's of examples in the service defs where we do it (e.g. yarnuiv2)
> 
> Best,
> Sandeep
> 
> On Thu, Jul 18, 2019 at 3:12 PM Praveen krishnamoorthy Ravikumar <pr334@njit.edu <ma...@njit.edu>> wrote:
> Greetings,
> 
> I’m working currently on enabling Amazon EMR debugging UIs running in private subnet via apache Knox. With the steps defined in the documentation I was able to install Knox and access YARN/SparkHistory/Ganglia UIs, which was amazing. But I’m facing issues on accessing certain links particularly the redirection links in NODE MANAGER UI.
> 
> My first question, Does the Knox supports redirection internally ?. For instance I was trying to access the container log message in the node manager UI (NODEMANAGERUI -> Local Logs [under tools] -> containers/ and was getting blocked with the browser URL - http://ip-10X.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042 <http://ip-10x.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042>
> 
> 
> What I noticed was the links that returning Http 3xx are getting broken.
> 
> Could anyone please help me resolving this issue ?
> 
> Thanks,
> Praveen.
> 
> Log message in gateway.log :
> —————————————
> 
> 2019-07-18 18:41:50,960 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /node/logs/containers
> 
> 2019-07-18 18:41:50,963 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://cas-query-sandbox-.emr.us-east-1.opse.c1.com:443/gateway/cto/node/logs/containers?host=ip-10-X.us-east-1.opse.c1.com&port=8042 <https://cas-query-sandbox-sc-866316499622-pp-oeobt24ki6aja.emr.us-east-1.opse.c1.vanguard.com/gateway/cto/node/logs/containers?host=ip-10-156-232-245.us-east-1.opse.c1.vanguard.com&port=8042>, direction: IN via implicit rule: NODEUI/logs/containers to URL: http://ip-10-156X.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042 <http://ip-10-156x.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042>
> 
> 2019-07-18 18:41:50,964 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://cas-query-sandbox.emr.us-east-1.opse.c1.com/gateway/cto/node/logs/?host=ip-X.us-east-1.opse.c1.com&port=8042 <https://cas-query-sandbox.emr.us-east-1.opse.c1.com/gateway/cto/node/logs/?host=ip-X.us-east-1.opse.c1.com&port=8042>, direction: IN via implicit rule: NODEUI/logs to URL: http://ip-10X.us-east-1.opse.c1.com:8042/logs/ <http://ip-10x.us-east-1.opse.c1.com:8042/logs/>
> 
> 2019-07-18 18:41:50,964 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET http://ipX.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.com&port=8042 <http://ipx.us-east-1.opse.c1.vanguard.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.vanguard.com&port=8042>
> 
> 2019-07-18 18:41:50,967 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 302
> 
> 2019-07-18 18:41:50,968 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit character set UTF-8 for entity of type text/plain
> 
> 2019-07-18 18:41:50,968 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound response entity content type: text/plain; charset=utf-8
> 
>

Re: Does KNOX support redirection links

Posted by Sandeep Moré <mo...@gmail.com>.
Hello Praveen,

If the UI is using Location header to redirect (which is most of the times)
then Knox can do it, there are lot's of examples in the service defs where
we do it (e.g. yarnuiv2)

Best,
Sandeep

On Thu, Jul 18, 2019 at 3:12 PM Praveen krishnamoorthy Ravikumar <
pr334@njit.edu> wrote:

> Greetings,
>
> I’m working currently on enabling Amazon EMR debugging UIs running in
> private subnet via apache Knox. With the steps defined in the documentation
> I was able to install Knox and access YARN/SparkHistory/Ganglia UIs, which
> was amazing. But I’m facing issues on accessing certain links particularly
> the redirection links in NODE MANAGER UI.
>
> My first question, Does the Knox supports redirection internally ?. For
> instance I was trying to access the container log message in the node
> manager UI (NODEMANAGERUI -> Local Logs [under tools] -> containers/ and
> was getting blocked with the browser URL -
> http://ip-10X.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042
> <http://ip-10x.us-east-1.opse.c1.com:8042/logs/containers/?host=ip-10X.us-east-1.opse.c1.com&port=8042>
>
>
> What I noticed was the links that returning Http 3xx are getting broken.
>
> Could anyone please help me resolving this issue ?
>
> Thanks,
> Praveen.
>
> Log message in gateway.log :
> —————————————
>
> 2019-07-18 18:41:50,960 DEBUG knox.gateway
> (GatewayFilter.java:doFilter(119)) - Received request: GET
> /node/logs/containers
>
> 2019-07-18 18:41:50,963 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://cas-query-sandbox-.emr.us-east-1.opse.c1.com:443/gateway/cto/node/logs/containers?host=ip-10-X.us-east-1.opse.c1.com&port=8042
> <https://cas-query-sandbox-sc-866316499622-pp-oeobt24ki6aja.emr.us-east-1.opse.c1.vanguard.com/gateway/cto/node/logs/containers?host=ip-10-156-232-245.us-east-1.opse.c1.vanguard.com&port=8042>,
> direction: IN via implicit rule: NODEUI/logs/containers to URL:
> http://ip-10-156X.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042
> <http://ip-10-156x.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10-156X.us-east-1.opse.c1.com&port=8042>
>
> 2019-07-18 18:41:50,964 DEBUG knox.gateway
> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL:
> https://cas-query-sandbox.emr.us-east-1.opse.c1.com/gateway/cto/node/logs/?host=ip-X.us-east-1.opse.c1.com&port=8042,
> direction: IN via implicit rule: NODEUI/logs to URL:
> http://ip-10X.us-east-1.opse.c1.com:8042/logs/
> <http://ip-10x.us-east-1.opse.c1.com:8042/logs/>
>
> 2019-07-18 18:41:50,964 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET
> http://ipX.us-east-1.opse.c1.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.com&port=8042
> <http://ipx.us-east-1.opse.c1.vanguard.com:8042/logs/containers?host=ip-10X.us-east-1.opse.c1.vanguard.com&port=8042>
>
> 2019-07-18 18:41:50,967 DEBUG knox.gateway
> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response
> status: *302*
>
> 2019-07-18 18:41:50,968 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit
> character set UTF-8 for entity of type text/plain
>
> 2019-07-18 18:41:50,968 DEBUG knox.gateway
> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound
> response entity content type: text/plain; charset=utf-8
>
>
>