You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@knox.apache.org by "Ravikumar, Praveen Krishnamoorthy" <rp...@amazon.com> on 2019/07/10 19:10:32 UTC
Sparks application list not getting loaded on accessing sparkUI via
Knox - Need your help
Greetings Everyone,
Currently I’m helping a customer to set up Knox to enable EMR UI’s (Yarn/SparkHistoryUI). As of now, I was able to install knox and access UIs through the browser. But I’m facing a weird issue on accessing SparkHistoryUI. The list of spark applications are not getting loaded in the web page. I have shared log message in gateway.log and also the browser message below.
Log Message :
2019-07-10 16:42:38,230 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /sparkhistory/static/vis.map
2019-07-10 16:42:38,231 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://cas-query-sandbox-sc--pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/static/vis.map, direction: IN via implicit rule: SPARKHISTORYUI/sparkhistory/inbound/query to URL: http://ip-XX.us-east-1.opse.c1.vanguard.com:18080/static/vis.map
2019-07-10 16:42:38,232 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET http://ip-XX.us-east-1.opse.c1.vanguard.com:18080/static/vis.map
2019-07-10 16:42:38,237 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 404
2019-07-10 16:42:38,238 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit character set ISO-8859-1 for entity of type text/html
2019-07-10 16:42:38,238 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound response entity content type: text/html; charset=iso-8859-1
2019-07-10 16:43:12,953 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /sparkhistory/https://cas-query-sandbox-sc--pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/api/v1/applications
Browser Message:
Failed to load resource: the server responded with a status of 404 (Not Found) - https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/api/v1/applications
For Verifying this, I tried in my personal amazon account and but I didn’t face this issue. I was able to see the spark apps getting loaded while accessing the sparkhistory UI. Here Is the log message.
Log message:
2019-07-10 17:44:10,967 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /sparkhistory/https://ec2-3-218-241-244.compute-1.amazonaws.com:8446/gateway/gate2/sparkhistory/api/v1/applications
2019-07-10 17:44:10,969 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://ec2-3-218-241-244.compute-1.amazonaws.com:8446/gateway/gate2/sparkhistory/https:/ec2-3-218-241-244.compute-1.amazonaws.com:8446/gateway/gate2/sparkhistory/api/v1/applications?limit=2147483647, direction: IN via implicit rule: SPARKHISTORYUI/sparkhistory/inbound/query to URL: http://ip-172-31-8-20.ec2.internal:18080/api/v1/applications?limit=2147483647
2019-07-10 17:44:10,969 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(163)) - Rewrote URL: /gateway/gate2, direction: IN via explicit rule: SPARKHISTORYUI/sparkhistory/outbound/rqheaders/xfc to URL: gateway/gate2/sparkhistoryui
2019-07-10 17:44:10,969 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET http://ip-172-31-8-20.ec2.internal:18080/api/v1/applications?limit=2147483647&user.name=admin
2019-07-10 17:44:11,013 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 200
2019-07-10 17:44:11,014 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(199)) - Using default character set UTF-8 for entity of type application/json
2019-07-10 17:44:11,014 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound response entity content type: application/json; charset=UTF-8
I suspect, the browser link that I’m getting in the customer environment is not getting mapped to the implicit rule IN via implicit rule: SPARKHISTORYUI/sparkhistory/inbound/query. I’m clueless of what needs to be changed for getting it mapped to the rule.
Could anyone please help me in fixing this, It would help me greatly. I can also give you more information if needed.
Thanks,
Praveen.
Re: Sparks application list not getting loaded on accessing sparkUI
via Knox - Need your help
Posted by "Ravikumar, Praveen Krishnamoorthy" <rp...@amazon.com>.
Sorry this is the Browser message:
Browser Message:
Failed to load resource: the server responded with a status of 404 (Not Found) - https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com/gateway/cto2/sparkhistory/https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/api/v1/applications
Note – Port number 443 is missing in the 1st occurring url address. (:443 was not appearing)
https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/api/v1/applications<https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/https:/cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/api/v1/applications>
Thanks,
Praveen.
From: "Ravikumar, Praveen Krishnamoorthy" <rp...@amazon.com>
Date: Wednesday, July 10, 2019 at 3:10 PM
To: "dev@knox.apache.org" <de...@knox.apache.org>, "user@knox.apache.org" <us...@knox.apache.org>
Subject: Sparks application list not getting loaded on accessing sparkUI via Knox - Need your help
Greetings Everyone,
Currently I’m helping a customer to set up Knox to enable EMR UI’s (Yarn/SparkHistoryUI). As of now, I was able to install knox and access UIs through the browser. But I’m facing a weird issue on accessing SparkHistoryUI. The list of spark applications are not getting loaded in the web page. I have shared log message in gateway.log and also the browser message below.
Log Message :
2019-07-10 16:42:38,230 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /sparkhistory/static/vis.map
2019-07-10 16:42:38,231 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://cas-query-sandbox-sc--pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/static/vis.map, direction: IN via implicit rule: SPARKHISTORYUI/sparkhistory/inbound/query to URL: http://ip-XX.us-east-1.opse.c1.vanguard.com:18080/static/vis.map
2019-07-10 16:42:38,232 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET http://ip-XX.us-east-1.opse.c1.vanguard.com:18080/static/vis.map
2019-07-10 16:42:38,237 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 404
2019-07-10 16:42:38,238 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(203)) - Using explicit character set ISO-8859-1 for entity of type text/html
2019-07-10 16:42:38,238 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound response entity content type: text/html; charset=iso-8859-1
2019-07-10 16:43:12,953 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /sparkhistory/https://cas-query-sandbox-sc--pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/api/v1/applications
Browser Message:
Failed to load resource: the server responded with a status of 404 (Not Found) - https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/https://cas-query-sandbox-sc-pp-fny2uhoy4ufgy.emr.us-east-1.opse.c1.com:443/gateway/cto2/sparkhistory/api/v1/applications
For Verifying this, I tried in my personal amazon account and but I didn’t face this issue. I was able to see the spark apps getting loaded while accessing the sparkhistory UI. Here Is the log message.
Log message:
2019-07-10 17:44:10,967 DEBUG knox.gateway (GatewayFilter.java:doFilter(119)) - Received request: GET /sparkhistory/https://ec2-3-218-241-244.compute-1.amazonaws.com:8446/gateway/gate2/sparkhistory/api/v1/applications
2019-07-10 17:44:10,969 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: https://ec2-3-218-241-244.compute-1.amazonaws.com:8446/gateway/gate2/sparkhistory/https:/ec2-3-218-241-244.compute-1.amazonaws.com:8446/gateway/gate2/sparkhistory/api/v1/applications?limit=2147483647, direction: IN via implicit rule: SPARKHISTORYUI/sparkhistory/inbound/query to URL: http://ip-172-31-8-20.ec2.internal:18080/api/v1/applications?limit=2147483647
2019-07-10 17:44:10,969 DEBUG knox.gateway (UrlRewriteProcessor.java:rewrite(163)) - Rewrote URL: /gateway/gate2, direction: IN via explicit rule: SPARKHISTORYUI/sparkhistory/outbound/rqheaders/xfc to URL: gateway/gate2/sparkhistoryui
2019-07-10 17:44:10,969 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET http://ip-172-31-8-20.ec2.internal:18080/api/v1/applications?limit=2147483647&user.name=admin
2019-07-10 17:44:11,013 DEBUG knox.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 200
2019-07-10 17:44:11,014 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(199)) - Using default character set UTF-8 for entity of type application/json
2019-07-10 17:44:11,014 DEBUG knox.gateway (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound response entity content type: application/json; charset=UTF-8
I suspect, the browser link that I’m getting in the customer environment is not getting mapped to the implicit rule IN via implicit rule: SPARKHISTORYUI/sparkhistory/inbound/query. I’m clueless of what needs to be changed for getting it mapped to the rule.
Could anyone please help me in fixing this, It would help me greatly. I can also give you more information if needed.
Thanks,
Praveen.