You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Robert Metzger <rm...@apache.org> on 2017/06/01 16:08:18 UTC
Re: Problems submitting Flink to Yarn with Kerberos
Can you check the logs of the JobManager? (maybe in DEBUG level), to see if
you see something that tries to establish a connection with it?
Are you sure you are properly authenticated to access the JM?
On Tue, May 30, 2017 at 4:41 PM, Dominique Rondé <
dominique.ronde@allsecur.de> wrote:
> Hi Gordon,
>
> we use Flink Flink 1.2.0 bundled with Hadoop 2.6 and Scala 2.11 build on
> 2017-02-02.
>
> Cheers
>
> Dominique
>
> Am 30.05.2017 um 16:31 schrieb Tzu-Li (Gordon) Tai:
>
> Hi Dominique,
>
> Could you tell us the version / build commit of Flink that you’re using?
>
> Cheers,
> Gordon
>
>
> On 30 May 2017 at 4:29:08 PM, Dominique Rondé (dominique.ronde@allsecur.de)
> wrote:
>
> Hi folks,
>
> I just become into the need to bring Flink into a yarn system, that is
> configured with kerberos. According to the documentation, I changed the
> flink.conf.yaml like that:
>
> security.kerberos.login.use-ticket-cache: true
> security.kerberos.login.contexts: Client
>
> I know that providing a keytab is the prefered, but I have to do a special
> request to receive one. ;-)
>
> After startup, the provisionent is stopped by this error:
>
> 2017-05-30 16:16:48,684 INFO org.apache.flink.yarn.
> YarnClusterClient - Waiting until all TaskManagers
> have connected
> Waiting until all TaskManagers have connected
> 2017-05-30 16:16:48,685 INFO org.apache.flink.yarn.
> YarnClusterClient - Starting client actor system.
> 2017-05-30 16:16:52,099 WARN org.apache.flink.runtime.net.
> ConnectionUtils - Could not connect to
> lfrar255.srv.allianz/10.17.24.162:56659. Selecting a local address using
> heuristics.
> 2017-05-30 16:16:52,473 INFO akka.event.slf4j.Slf4jLogger
> - Slf4jLogger started
> 2017-05-30 16:16:52,512 INFO Remoting
> - Starting remoting
> 2017-05-30 16:16:52,670 INFO Remoting
> - Remoting started; listening on
> addresses :[akka.tcp://flink@sla09037.srv.allianz:34579]
> Exception in thread "main" java.lang.RuntimeException: Unable to get
> ClusterClient status from Application Client
> at org.apache.flink.yarn.YarnClusterClient.getClusterStatus(
> YarnClusterClient.java:248)
> at org.apache.flink.yarn.YarnClusterClient.
> waitForClusterToBeReady(YarnClusterClient.java:520)
> at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(
> FlinkYarnSessionCli.java:660)
> at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(
> FlinkYarnSessionCli.java:476)
> at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(
> FlinkYarnSessionCli.java:473)
> at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(
> HadoopSecurityContext.java:43)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1656)
> at org.apache.flink.runtime.security.HadoopSecurityContext.
> runSecured(HadoopSecurityContext.java:40)
> at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(
> FlinkYarnSessionCli.java:473)
> Caused by: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException:
> Could not retrieve the leader gateway
> at org.apache.flink.runtime.util.LeaderRetrievalUtils.
> retrieveLeaderGateway(LeaderRetrievalUtils.java:141)
> at org.apache.flink.client.program.ClusterClient.
> getJobManagerGateway(ClusterClient.java:691)
> at org.apache.flink.yarn.YarnClusterClient.getClusterStatus(
> YarnClusterClient.java:242)
> ... 10 more
> Caused by: java.util.concurrent.TimeoutException: Futures timed out after
> [10000 milliseconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.
> scala:219)
> at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.
> scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(
> package.scala:190)
> at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(
> BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:190)
> at scala.concurrent.Await.result(package.scala)
> at org.apache.flink.runtime.util.LeaderRetrievalUtils.
> retrieveLeaderGateway(LeaderRetrievalUtils.java:139)
> ... 12 more
> 2017-05-30 16:17:02,690 INFO org.apache.flink.yarn.
> YarnClusterClient - Shutting down YarnClusterClient
> from the client shutdown hook
> 2017-05-30 16:17:02,691 INFO org.apache.flink.yarn.
> YarnClusterClient - Disconnecting YarnClusterClient
> from ApplicationMaster
> 2017-05-30 16:17:03,693 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator
> - Shutting down remote daemon.
> 2017-05-30 16:17:03,696 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator
> - Remote daemon shut down; proceeding with flushing remote transports.
> 2017-05-30 16:17:03,744 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator
> - Remoting shut down.
>
> Has anyone an idea what is going wrong?
>
> Best wished
>
> Dominique
>
>
>
Re: Problems submitting Flink to Yarn with Kerberos
Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.
Hi Dominique,
Had another quick look at the error trace you provided, the problem doesn’t seem to be related to Kerberos authentication.
For some reason the JobManager simply isn’t reachable from the client, as Robert has pointed out. There should be some clue about this in the JM logs.
On 1 June 2017 at 6:08:42 PM, Robert Metzger (rmetzger@apache.org) wrote:
Can you check the logs of the JobManager? (maybe in DEBUG level), to see if you see something that tries to establish a connection with it?
Are you sure you are properly authenticated to access the JM?
On Tue, May 30, 2017 at 4:41 PM, Dominique Rondé <do...@allsecur.de> wrote:
Hi Gordon,
we use Flink Flink 1.2.0 bundled with Hadoop 2.6 and Scala 2.11 build on 2017-02-02.
Cheers
Dominique
Am 30.05.2017 um 16:31 schrieb Tzu-Li (Gordon) Tai:
Hi Dominique,
Could you tell us the version / build commit of Flink that you’re using?
Cheers,
Gordon
On 30 May 2017 at 4:29:08 PM, Dominique Rondé (dominique.ronde@allsecur.de) wrote:
Hi folks,
I just become into the need to bring Flink into a yarn system, that is configured with kerberos. According to the documentation, I changed the flink.conf.yaml like that:
security.kerberos.login.use-ticket-cache: true
security.kerberos.login.contexts: Client
I know that providing a keytab is the prefered, but I have to do a special request to receive one. ;-)
After startup, the provisionent is stopped by this error:
2017-05-30 16:16:48,684 INFO org.apache.flink.yarn.YarnClusterClient - Waiting until all TaskManagers have connected
Waiting until all TaskManagers have connected
2017-05-30 16:16:48,685 INFO org.apache.flink.yarn.YarnClusterClient - Starting client actor system.
2017-05-30 16:16:52,099 WARN org.apache.flink.runtime.net.ConnectionUtils - Could not connect to lfrar255.srv.allianz/10.17.24.162:56659. Selecting a local address using heuristics.
2017-05-30 16:16:52,473 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2017-05-30 16:16:52,512 INFO Remoting - Starting remoting
2017-05-30 16:16:52,670 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@sla09037.srv.allianz:34579]
Exception in thread "main" java.lang.RuntimeException: Unable to get ClusterClient status from Application Client
at org.apache.flink.yarn.YarnClusterClient.getClusterStatus(YarnClusterClient.java:248)
at org.apache.flink.yarn.YarnClusterClient.waitForClusterToBeReady(YarnClusterClient.java:520)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:660)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:476)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:473)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:473)
Caused by: org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Could not retrieve the leader gateway
at org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderGateway(LeaderRetrievalUtils.java:141)
at org.apache.flink.client.program.ClusterClient.getJobManagerGateway(ClusterClient.java:691)
at org.apache.flink.yarn.YarnClusterClient.getClusterStatus(YarnClusterClient.java:242)
... 10 more
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:190)
at scala.concurrent.Await.result(package.scala)
at org.apache.flink.runtime.util.LeaderRetrievalUtils.retrieveLeaderGateway(LeaderRetrievalUtils.java:139)
... 12 more
2017-05-30 16:17:02,690 INFO org.apache.flink.yarn.YarnClusterClient - Shutting down YarnClusterClient from the client shutdown hook
2017-05-30 16:17:02,691 INFO org.apache.flink.yarn.YarnClusterClient - Disconnecting YarnClusterClient from ApplicationMaster
2017-05-30 16:17:03,693 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon.
2017-05-30 16:17:03,696 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remote daemon shut down; proceeding with flushing remote transports.
2017-05-30 16:17:03,744 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Remoting shut down.
Has anyone an idea what is going wrong?
Best wished
Dominique