You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Scott Kidder (JIRA)" <ji...@apache.org> on 2017/06/27 23:54:00 UTC
[jira] [Created] (FLINK-7022) Flink Job Manager Scheduler & Web
Frontend out of sync when Zookeeper is unavailable on startup
Scott Kidder created FLINK-7022:
-----------------------------------
Summary: Flink Job Manager Scheduler & Web Frontend out of sync when Zookeeper is unavailable on startup
Key: FLINK-7022
URL: https://issues.apache.org/jira/browse/FLINK-7022
Project: Flink
Issue Type: Bug
Components: JobManager
Affects Versions: 1.2.1, 1.3.0, 1.2.0
Environment: Kubernetes cluster running:
* Flink 1.3.0 Job Manager & Task Manager on Java 8u131
* Zookeeper 3.4.10 cluster with 3 nodes
Reporter: Scott Kidder
h2. Problem
Flink Job Manager web frontend is permanently unavailable if one or more Zookeeper nodes are unresolvable during startup. The job scheduler eventually recovers and assigns jobs to task managers, but the web frontend continues to respond with an HTTP 503 and the following message:
{noformat}Service temporarily unavailable due to an ongoing leader election. Please refresh.{noformat}
h2. Expected Behavior
Once Flink is able to interact with Zookeeper successfully, all aspects of the Job Manager (job scheduling & the web frontend) should be available.
h2. Environment Details
We're running Flink and Zookeeper in Kubernetes on CoreOS. CoreOS can run in a configuration that automatically detects and applies operating system updates. We have a Zookeeper node running on the same CoreOS instance as Flink. It's possible that the Zookeeper node will not yet be started when the Flink components are started. This could cause hostname resolution of the Zookeeper nodes to fail.
h3. Flink Task Manager Logs
{noformat}
2017-06-27 15:38:47,161 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.host, localhost
2017-06-27 15:38:47,161 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.port, 8125
2017-06-27 15:38:47,162 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.interval, 10 SECONDS
2017-06-27 15:38:47,254 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend, filesystem
2017-06-27 15:38:47,254 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.fs.checkpointdir, hdfs://hdfs:8020/flink/checkpoints
2017-06-27 15:38:47,255 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, hdfs://hdfs:8020/flink/savepoints
2017-06-27 15:38:47,255 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.mode, zookeeper
2017-06-27 15:38:47,256 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.quorum, zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
2017-06-27 15:38:47,256 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.storageDir, hdfs://hdfs:8020/flink/recovery
2017-06-27 15:38:47,256 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.jobmanager.port, 6123
2017-06-27 15:38:47,257 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 41479
2017-06-27 15:38:47,357 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-06-27 15:38:47,366 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager with high-availability
2017-06-27 15:38:47,366 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.jobmanager.port' instead of proper key 'high-availability.jobmanager.port'
2017-06-27 15:38:47,452 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager on flink:6123 with execution mode CLUSTER
2017-06-27 15:38:47,549 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink
2017-06-27 15:38:47,549 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2017-06-27 15:38:47,549 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
2017-06-27 15:38:47,549 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 1024
2017-06-27 15:38:47,549 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-06-27 15:38:47,549 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false
2017-06-27 15:38:47,550 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2017-06-27 15:38:47,550 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081
2017-06-27 15:38:47,550 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporters, statsd
2017-06-27 15:38:47,550 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.class, org.apache.flink.metrics.statsd.StatsDReporter
2017-06-27 15:38:47,551 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.host, localhost
2017-06-27 15:38:47,551 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.port, 8125
2017-06-27 15:38:47,551 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.interval, 10 SECONDS
2017-06-27 15:38:47,551 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend, filesystem
2017-06-27 15:38:47,551 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.fs.checkpointdir, hdfs://hdfs:8020/flink/checkpoints
2017-06-27 15:38:47,552 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.savepoints.dir, hdfs://hdfs:8020/flink/savepoints
2017-06-27 15:38:47,552 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.mode, zookeeper
2017-06-27 15:38:47,552 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.quorum, zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
2017-06-27 15:38:47,552 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.zookeeper.storageDir, hdfs://hdfs:8020/flink/recovery
2017-06-27 15:38:47,552 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: recovery.jobmanager.port, 6123
2017-06-27 15:38:47,552 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 41479
2017-06-27 15:38:48,055 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to root (auth:SIMPLE)
2017-06-27 15:38:48,664 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system reachable at flink:6123
2017-06-27 15:38:50,955 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2017-06-27 15:38:51,252 INFO Remoting - Starting remoting
2017-06-27 15:38:52,679 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@flink:6123]
2017-06-27 15:38:52,758 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-06-27 15:38:52,761 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-06-27 15:38:52,764 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir'
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, flink
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 1024
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2017-06-27 15:38:52,854 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081
2017-06-27 15:38:52,864 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporters, statsd
2017-06-27 15:38:52,865 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.class, org.apache.flink.metrics.statsd.StatsDReporter
2017-06-27 15:38:52,865 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.host, localhost
2017-06-27 15:38:52,865 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.port, 8125
2017-06-27 15:38:52,865 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: metrics.reporter.statsd.interval, 10 SECONDS
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at org.apache.flink.shaded.org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:262)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.start(ConnectionState.java:109)
at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:191)
at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
at org.apache.flink.runtime.util.ZooKeeperUtils.startCuratorFramework(ZooKeeperUtils.java:128)
at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:96)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2047)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:38:59,160 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager web frontend
2017-06-27 15:38:59,257 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager log file: /usr/local/flink-1.3.0/log/flink--jobmanager-0-flink-jobmanager-3380372638-1q7jb.log
2017-06-27 15:38:59,257 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager stdout file: /usr/local/flink-1.3.0/log/flink--jobmanager-0-flink-jobmanager-3380372638-1q7jb.out
2017-06-27 15:38:59,257 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-252afcf4-d41d-4095-a082-f6ce5176c2f5 for the web interface files
2017-06-27 15:38:59,257 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-2ca2cadf-a1b6-44af-9510-9c523a422022 for web frontend JAR file uploads
2017-06-27 15:39:01,060 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0:0:0:0:0:0:0:0:8081
2017-06-27 15:39:01,060 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor
2017-06-27 15:39:01,253 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-1f49aadd-0a7d-45d1-8fdc-fc2167ca93d5
2017-06-27 15:39:01,257 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:41479 - max concurrent requests: 50 - max backlog: 1000
2017-06-27 15:39:01,851 INFO org.apache.flink.runtime.metrics.MetricRegistry - Configuring StatsDReporter with {interval=10 SECONDS, port=8125, host=localhost, class=org.apache.flink.metrics.statsd.StatsDReporter}.
2017-06-27 15:39:01,948 INFO org.apache.flink.metrics.statsd.StatsDReporter - Configured StatsDReporter with {host:localhost, port:8125}
2017-06-27 15:39:01,949 INFO org.apache.flink.runtime.metrics.MetricRegistry - Periodically reporting metrics in intervals of 10 SECONDS for reporter statsd of type org.apache.flink.metrics.statsd.StatsDReporter.
2017-06-27 15:39:02,050 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/archive
2017-06-27 15:39:02,059 WARN org.apache.flink.configuration.Configuration - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper key 'high-availability.storageDir'
2017-06-27 15:39:17,252 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection timed out for connection string (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181) and timeout (15000) / elapsed (18395)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:109)
at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:469)
at org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.<init>(ZooKeeperSubmittedJobGraphStore.java:116)
at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:39:37,448 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:40:07,457 WARN org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection attempt unsuccessful after 68603 (greater than max timeout of 60000). Resetting connection and trying again with a new connection.
2017-06-27 15:40:07,457 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:40:07,555 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl - Ensure path threw exception
java.net.UnknownHostException: zookeeper-1.zookeeper: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
at org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at org.apache.flink.shaded.org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
at org.apache.flink.shaded.org.apache.curator.HandleHolder.closeAndReset(HandleHolder.java:77)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:261)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:221)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
at org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:109)
at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:469)
at org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.<init>(ZooKeeperSubmittedJobGraphStore.java:116)
at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:40:22,566 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection timed out for connection string (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181) and timeout (15000) / elapsed (15108)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:40:42,575 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:41:02,684 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState - Connection timed out for connection string (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181) and timeout (15000) / elapsed (55226)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.flink.shaded.org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:156)
at org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.flink.shaded.org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:149)
at org.apache.flink.shaded.org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:102)
at org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.<init>(ZooKeeperSubmittedJobGraphStore.java:117)
at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
at scala.util.Try$.apply(Try.scala:192)
at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:41:02,684 INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181 sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:41:02,803 WARN org.apache.zookeeper.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1381454376626202001.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2017-06-27 15:41:02,804 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState - Authentication failed
2017-06-27 15:41:02,806 INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server ip-10-2-8-5.ec2.internal/10.2.8.5:2181
...
2017-06-27 16:00:51,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Try to restart or fail the job (022d8149808dd3297a8a7275a1fd3d6b) if no longer possible.
2017-06-27 16:00:51,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job (022d8149808dd3297a8a7275a1fd3d6b) switched from state FAILING to RESTARTING.
2017-06-27 16:00:51,490 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Restarting the job (022d8149808dd3297a8a7275a1fd3d6b).
2017-06-27 16:00:51,490 INFO org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter - Delaying retry of job execution for 10000 ms ...
2017-06-27 16:00:58,252 INFO org.apache.flink.runtime.jobmanager.JobManager - Task Manager Registration but not connected to ResourceManager
2017-06-27 16:00:58,254 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at flink-taskmanager-3116622558-zmmwq (akka.tcp://flink@10.2.8.11:6122/user/taskmanager) as 2a058f00bd1e25f44c1cb8f3e5dd726f. Current number of registered hosts is 1. Current number of alive task slots is 2.
2017-06-27 16:00:58,453 INFO org.apache.flink.runtime.jobmanager.JobManager - Task Manager Registration but not connected to ResourceManager
2017-06-27 16:01:01,491 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job (022d8149808dd3297a8a7275a1fd3d6b) switched from state RESTARTING to CREATED.
2017-06-27 16:01:01,491 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Recovering checkpoints from ZooKeeper.
2017-06-27 16:01:01,645 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found 1 checkpoints in ZooKeeper.
2017-06-27 16:01:01,645 INFO org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Trying to retrieve checkpoint 502.
2017-06-27 16:01:01,660 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Restoring from latest valid checkpoint: Checkpoint 502 @ 1498577858587 for 022d8149808dd3297a8a7275a1fd3d6b.
2017-06-27 16:01:01,661 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - No master state to restore
2017-06-27 16:01:01,661 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job (022d8149808dd3297a8a7275a1fd3d6b) switched from state CREATED to RUNNING.
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)