You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gearpump.apache.org by titikakatoo <gi...@git.apache.org> on 2017/10/11 12:06:25 UTC
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
GitHub user titikakatoo opened a pull request:
https://github.com/apache/incubator-gearpump/pull/231
[GEARPUMP-355] Fix YarnAppMaster address resolution in a kerberized H…
…adoop/Yarn set-up
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
- [ ] Make sure the commit message is formatted like:
`[GEARPUMP-<Jira issue #>] Meaningful description of pull request`
- [ ] Make sure tests pass via `sbt clean test`.
- [ ] Make sure old documentation affected by the pull request has been updated and new documentation added for new functionality.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/titikakatoo/incubator-gearpump yarn_spnego_authentication
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-gearpump/pull/231.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #231
----
commit 33093d6e6c15a2250d488ec356b6a0093f1b6215
Author: Timea Magyar <ti...@etas.com>
Date: 2017-10-11T12:05:10Z
[GEARPUMP-355] Fix YarnAppMaster address resolution in a kerberized Hadoop/Yarn set-up
----
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r145602797
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +64,38 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getOriginalTrackingUrl}" + "supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection = connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+ connection.connect()
--- End diff --
How will `IOException`thrown on connect be handled ?
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
R: @huafengw
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
merging. Thanks.
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-gearpump/pull/231
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r145601399
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +64,38 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getOriginalTrackingUrl}" + "supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection = connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+ connection.connect()
+ val status = connection.getResponseCode
+
+ var stream: java.io.InputStream = connection.getErrorStream
+ if(stream == null) {
+ stream = connection.getInputStream
+ }
+ if(status == 200) {
--- End diff --
fix style: need a space before `(`
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r145602983
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +64,38 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getOriginalTrackingUrl}" + "supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection = connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+ connection.connect()
+ val status = connection.getResponseCode
+
+ var stream: java.io.InputStream = connection.getErrorStream
--- End diff --
Shall we check `status` first since otherwise there is no need to check `stream` ?
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
@titikakatoo Added some comments. BTW, can upgrade http client version fix the issue ?
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by titikakatoo <gi...@git.apache.org>.
Github user titikakatoo commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
@manuzhang upgrading the http client version might (in theory) also fix the issue, have not tested it though.
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
Hi @titikakatoo, thanks for your response. I'd like to know how you set up your cluster. I just got a single-node secured yarn cluster but found the command you posted goes well.
So how many nodes are there in your cluster and where do you execute the yarnClient command, just one of the node among them? And also, is there any special config I should notice?
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by titikakatoo <gi...@git.apache.org>.
Github user titikakatoo commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
@huafengw
Answering your second question: this patch is to some extent related to #230 . The second patch is needed to correctly resolve the Application Master tracking url in any case, in a proxied and non-proxied scenario as well. Especially in a secured hadoop set-up the uri will be proxied.
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by titikakatoo <gi...@git.apache.org>.
Github user titikakatoo commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146003668
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection = connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+
+ try {
+ connection.connect()
+ } catch {
+ case e: IOException =>
+ LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+ }
+
+ val status = connection.getResponseCode
+ if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+ } else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please make sure " +
+ s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
@manuzhang getTrackingURL was merged already in the master branch.
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
Hi @titikakatoo , thanks for your contribution!
Can you briefly describe how to reproduce the problem you found? I'm setting up a secured cluster and try to verify your pull request.
And BTW, does this pull request depend on #230?
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by titikakatoo <gi...@git.apache.org>.
Github user titikakatoo commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
Hi @huafengw, I haven't set the cluster up myself. It is one of our corporate CDH 5.11.1 Hadoop cluster shared across multiple departments, secured by Kerberos/Reverse Proxy/SSL. It is a 10 node cluster located in the company's data center and protected by a corporate firewall (Security Layer 4) and accessible only via application/edge servers with sufficient firewall rules to access cluster services.
The above cluster was set up by our corporate CI department in cooperation with Cloudera and it is mostly leaned on standard Cloudera 5 security guidelines, nothing special in this respect.
The **YarnClient/Gear** commands are all executed from one of our edge nodes, by an appropriate Kerberos principal.
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r145601401
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +64,38 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getOriginalTrackingUrl}" + "supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection = connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+ connection.connect()
+ val status = connection.getResponseCode
+
+ var stream: java.io.InputStream = connection.getErrorStream
+ if(stream == null) {
--- End diff --
fix style: need a space before `(`
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
I've tested the pr and no regression was found.
But it seems this change introduces some unused imports to the affected files. We'll merge it once they were resolved. @titikakatoo
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by titikakatoo <gi...@git.apache.org>.
Github user titikakatoo commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
@manuzhang , @huafengw removed unused imports.
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by codecov-io <gi...@git.apache.org>.
Github user codecov-io commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
# [Codecov](https://codecov.io/gh/apache/incubator-gearpump/pull/231?src=pr&el=h1) Report
> Merging [#231](https://codecov.io/gh/apache/incubator-gearpump/pull/231?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-gearpump/commit/175b08e64c3363a2da9c61e05dac5f01d1f8db2d?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #231 +/- ##
==========================================
+ Coverage 69.01% 69.03% +0.01%
==========================================
Files 191 191
Lines 6119 6119
Branches 358 358
==========================================
+ Hits 4223 4224 +1
+ Misses 1896 1895 -1
```
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by titikakatoo <gi...@git.apache.org>.
Github user titikakatoo commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146019076
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection = connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+
+ try {
+ connection.connect()
+ } catch {
+ case e: IOException =>
+ LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+ }
+
+ val status = connection.getResponseCode
+ if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+ } else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please make sure " +
+ s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
Done.
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r145602273
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +64,38 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getOriginalTrackingUrl}" + "supervisor-actor-path"
--- End diff --
missing "/" here
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
LGTM. @huafengw more comments ?
---
[GitHub] incubator-gearpump issue #231: [GEARPUMP-355] Fix YarnAppMaster address reso...
Posted by titikakatoo <gi...@git.apache.org>.
Github user titikakatoo commented on the issue:
https://github.com/apache/incubator-gearpump/pull/231
@huafengw
I can provide you the following sanitized extract from the logs when trying to use gearpump on a kerberized hadoop cluster without the patch:
> [HttpClient] Java version: 1.8.0_141
> [HttpClient] Java vendor: Oracle Corporation
> [HttpClient] Java class path: <classpath>
> [HttpClient] Operating system name: <>
> [HttpClient] Operating system architecture: <>
> [HttpClient] Operating system version: <>
> [HttpClient] SUN 1.8: <>
> [HttpClient] SunRsaSign 1.8: <>
> [HttpClient] SunEC 1.8: <>
> [HttpClient] SunJSSE 1.8: <>
> [HttpClient] SunJCE 1.8: <>
> [HttpClient] SunJGSS 1.8: Sun **(Kerberos v5, SPNEGO)**
> [HttpClient] SunSASL 1.8: <>
> [HttpClient] XMLDSig 1.8: <>
> [HttpClient] SunPCSC 1.8: <>
> [HttpClient] SaslPlainServer 1.0: SASL PLAIN Authentication Server
> [DefaultHttpParams] Set parameter http.useragent = **Jakarta Commons-HttpClient/3.1**
> [DefaultHttpParams] Set parameter http.protocol.version = HTTP/1.1
> [DefaultHttpParams] Set parameter http.connection-manager.class = class org.apache.commons.httpclient.SimpleHttpConnectionManager
> [DefaultHttpParams] Set parameter http.protocol.cookie-policy = default
> [DefaultHttpParams] Set parameter http.protocol.element-charset = US-ASCII
> [DefaultHttpParams] Set parameter http.protocol.content-charset = ISO-8859-1
> [DefaultHttpParams] Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@54f5f647
> [DefaultHttpParams] Set parameter http.dateparser.patterns = <>
> [AppMasterResolver] appMasterPath=https://<proxy_host>:<proxy_port>/proxy/<application_id>//supervisor-actor-path
> [HttpConnection] Open connection to <proxy_host>:<proxy_port>
> [header] >> "GET /proxy/<application_id>//supervisor-actor-path HTTP/1.1[\r][\n]"
> [HttpMethodBase] Adding Host request header
> [header] >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
> [header] >> "Host: <proxy_host>:<proxy_port>[\r][\n]"
> [header] >> "[\r][\n]"
> [header] << "HTTP/1.1 401 Authentication required[\r][\n]"
> [header] << "HTTP/1.1 401 Authentication required[\r][\n]"
> [header] << "Cache-Control: must-revalidate,no-cache,no-store[\r][\n]"
> [header] << "Date: Thu, 12 Oct 2017 09:06:05 GMT[\r][\n]"
> [header] << "Pragma: no-cache[\r][\n]"
> [header] << "Date: Thu, 12 Oct 2017 09:06:05 GMT[\r][\n]"
> [header] << "Pragma: no-cache[\r][\n]"
> [header] << "Content-Type: text/html; charset=iso-8859-1[\r][\n]"
> [header] << "X-FRAME-OPTIONS: SAMEORIGIN[\r][\n]"
> [header] << "WWW-Authenticate: Negotiate[\r][\n]"
> [header] << "Set-Cookie: hadoop.auth=; Path=/; Secure; HttpOnly[\r][\n]"
> [header] << "Content-Length: 1452[\r][\n]"
> [header] << "Server: Jetty(6.1.26.cloudera.4)[\r][\n]"
> [header] << "[\r][\n]"
> [CookieSpec] Unrecognized cookie attribute: name=HttpOnly, value=null
> [HttpMethodBase] Cookie accepted: "$Version=0; hadoop.auth=; $Path=/"
> [HttpMethodDirector] Authorization required
> [AuthChallengeProcessor] Supported authentication schemes in the order of preference: **[ntlm, digest, basic]**
> [AuthChallengeProcessor] Challenge for ntlm authentication scheme not available
> [AuthChallengeProcessor] Challenge for digest authentication scheme not available
> [AuthChallengeProcessor] Challenge for basic authentication scheme not available
> [HttpMethodDirector] **Unable to respond to any of these challenges: {negotiate=Negotiate}**
> [AppMasterResolver] Failed to connect YarnAppMaster(tried 1)... Fail to resolve AppMaster address, please make sure https://<proxy_host>:<proxy_port>/proxy/<application_id>/ is accessible...
Currently gearpump uses the apache http client version 3.1. This client only supports the authentication schemes [ntlm, digest, basic] (see logs). As you can further see in the logs: Unable to respond to any of these challenges: {negotiate=Negotiate}, a communication via kerberos spnego is not possible in this case.
After the patch trying to get the active configuration from YarnAppmaster produces the following sanitized logs:
> [Client] IPC Client (<>) connection to <host><port> from kerberos principal: starting, having connections 1
> [SaslRpcClient] reading next wrapped RPC packet
> [Client] IPC Client (<>) connection to <host><port> from kerberos principal sending #0
> [SaslRpcClient] wrapping token of length:<length>
> [SaslRpcClient] unwrapping token of length:<length>
> [Client] IPC Client (<>) connection to <host><port> from kerberos principal got value #0
> [ProtobufRpcEngine] Call: getApplicationReport took 224ms
> [AppMasterResolver$] appMasterPath=https://<host>:<port>/proxy/application_<id>/supervisor-actor-path
> [FileBasedKeyStoresFactory] CLIENT TrustStore: <client_trust_store>
> [ReloadingX509TrustManager] Loaded truststore '<client_trust_store>'
> [FileBasedKeyStoresFactory] CLIENT Loaded TrustStore: <client_trust_store>
> [URLConnectionFactory] open URL connection
> [AppMasterResolver$] **Successfully resolved AppMaster address: akka.tcp://GearpumpAM@<host>:<port>/user/appMaster**
> **ActiveConfig(Config<config>)**
> [RemoteActorRefProvider$RemotingTerminator] Shutting down remote daemon.
> [RemoteActorRefProvider$RemotingTerminator] Remote daemon shut down; proceeding with flushing remote transports.
> [RemoteActorRefProvider$RemotingTerminator] Remoting shut down.
---
[GitHub] incubator-gearpump pull request #231: [GEARPUMP-355] Fix YarnAppMaster addre...
Posted by manuzhang <gi...@git.apache.org>.
Github user manuzhang commented on a diff in the pull request:
https://github.com/apache/incubator-gearpump/pull/231#discussion_r146002316
--- Diff: experiments/yarn/src/main/scala/org/apache/gearpump/experiments/yarn/client/AppMasterResolver.scala ---
@@ -75,3 +65,40 @@ class AppMasterResolver(yarnClient: YarnClient, system: ActorSystem) {
result
}
}
+
+object AppMasterResolver {
+ val LOG = LogUtil.getLogger(getClass)
+
+ def resolveAppMasterAddress(report: ApplicationReport, system: ActorSystem): ActorRef = {
+ val appMasterPath = s"${report.getTrackingURL}/supervisor-actor-path"
+ LOG.info(s"appMasterPath=$appMasterPath")
+
+ val connectionFactory: URLConnectionFactory = URLConnectionFactory
+ .newDefaultURLConnectionFactory(new YarnConfiguration())
+ val url: URL = new URL(appMasterPath)
+ val connection: HttpURLConnection = connectionFactory.openConnection(url)
+ .asInstanceOf[HttpURLConnection]
+ connection.setInstanceFollowRedirects(true)
+
+ try {
+ connection.connect()
+ } catch {
+ case e: IOException =>
+ LOG.error(s"Failed to connect to AppMaster" + e.getMessage)
+ }
+
+ val status = connection.getResponseCode
+ if (status == 200) {
+ val stream: java.io.InputStream = connection.getInputStream
+ val response = IOUtils.toString(stream, StandardCharsets.UTF_8)
+ LOG.info("Successfully resolved AppMaster address: " + response)
+ connection.disconnect()
+ AkkaHelper.actorFor(system, response)
+ } else {
+ connection.disconnect()
+ throw new IOException("Fail to resolve AppMaster address, please make sure " +
+ s"${report.getOriginalTrackingUrl} is accessible...")
--- End diff --
should this be `getTrackingUrl` now ?
---