You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by miki haiat <mi...@gmail.com> on 2018/04/29 16:01:31 UTC
JM/TM Disassociated
Hi,
I have some simple elesticsearch sink on mesos with 6 TM ,
but for some reason after few hours JM/TM is Disassociated and killed
all other TM as well .
this is were everything collapse
2018-04-29 08:21:00,457 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated
for [5000] ms. Reason: [Disassociated]
2018-04-29 08:21:06,501 WARN
akka.remote.transport.netty.NettyTransport - Remote
connection to [null] failed with java.net.ConnectException: Connection
refused: /10.1.70.116:6123
2018-04-29 08:21:00,457 WARN akka.remote.ReliableDeliverySupervisor
- Association with remote system
[akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated
for [5000] ms. Reason: [Disassociated]
> 2018-04-29 08:21:06,501 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:06,503 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:16,501 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:16,504 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:26,500 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:26,501 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:31,124 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager: Old JobManager lost its leadership.
> 2018-04-29 08:21:31,124 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Cancelling all computations and discarding all cached data.
> 2018-04-29 08:21:31,125 INFO org.apache.flink.runtime.taskmanager.Task - Attempting to fail task externally Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae).
> 2018-04-29 08:21:31,127 INFO org.apache.flink.runtime.taskmanager.Task - Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae) switched from RUNNING to FAILED.
> java.lang.Exception: TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager: Old JobManager lost its leadership.
> at org.apache.flink.runtime.taskmanager.TaskManager.handleJobManagerDisconnect(TaskManager.scala:1073)
> at org.apache.flink.runtime.taskmanager.TaskManager.org$apache$flink$runtime$taskmanager$TaskManager$$handleJobManagerLeaderAddress(TaskManager.scala:1467)
> at org.apache.flink.runtime.taskmanager.TaskManager$$anonfun$handleMessage$1.applyOrElse(TaskManager.scala:277)
> at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:49)
> at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
> at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at org.apache.flink.runtime.taskmanager.TaskManager.aroundReceive(TaskManager.scala:121)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-04-29 08:21:31,129 INFO org.apache.flink.runtime.taskmanager.Task - Triggering cancellation of task code Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae).
> 2018-04-29 08:21:31,130 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Disassociating from JobManager
> 2018-04-29 08:21:31,141 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Shutting down BLOB cache
> 2018-04-29 08:21:31,145 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #17
> 2018-04-29 08:21:31,145 INFO org.apache.flink.runtime.taskmanager.Task - Freeing task resources for Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae).
> 2018-04-29 08:21:31,173 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #1
> 2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #2
> 2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #3
> 2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #4
> 2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #5
> 2018-04-29 08:21:31,172 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
> 2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #6
> 2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #7
> 2018-04-29 08:21:31,174 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #8
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #9
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #10
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #11
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #12
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #13
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #14
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #15
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #16
> 2018-04-29 08:21:31,175 ERROR org.apache.flink.streaming.connectors.elasticsearch5.shaded.org.jboss.netty.channel.socket.nio.AbstractNioSelector - Interrupted while wait for resources to be released #17
> 2018-04-29 08:21:31,211 INFO org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem streams are closed for task Source: Custom Source -> Sink: Unnamed (1/1) (8468d2974c12c77d4ddbbb4e48ed2eae) [FAILED]
> 2018-04-29 08:21:31,520 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:31,520 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:36,540 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:36,541 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:41,559 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:41,560 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:46,581 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:46,581 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:51,602 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:51,602 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:21:56,621 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:21:56,621 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:01,640 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:01,649 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:06,659 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:06,660 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:11,680 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:11,680 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:16,700 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:16,701 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:21,719 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:21,720 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:26,740 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:26,740 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:31,760 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:31,760 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:36,780 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:36,781 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:41,800 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:41,800 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:46,820 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:46,820 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:51,840 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:51,841 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:22:56,860 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:22:56,860 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:01,879 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:01,880 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:06,901 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:06,901 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:11,919 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:11,919 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:16,939 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:16,940 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:21,959 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:21,960 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:26,984 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:26,984 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:32,000 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:32,000 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:37,024 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:37,025 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:42,039 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:42,040 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:47,060 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:47,060 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:52,080 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:52,080 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:23:57,101 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:23:57,101 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:24:02,119 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:24:02,120 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:24:07,140 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:24:07,140 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:24:12,159 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:24:12,160 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:24:17,180 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:24:17,180 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:24:22,201 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:24:22,202 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:24:27,219 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 08:24:27,219 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 08:24:31,285 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Trying to register at JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)
> 2018-04-29 08:24:31,797 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Trying to register at JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager (attempt 2, timeout: 1000 milliseconds)
> 2018-04-29 08:24:32,240 ERROR akka.remote.Remoting - Association to [akka.tcp://flink@10.1.70.116:6123] with UID [1855709371] irrecoverably failed. Quarantining address.
> java.util.concurrent.TimeoutException: Delivery of system messages timed out and they were dropped.
> at akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoint.scala:346)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-04-29 08:24:32,817 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Trying to register at JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager (attempt 3, timeout: 2000 milliseconds)
> 2018-04-29 08:24:32,987 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Successful registration at JobManager (akka.tcp://flink@10.1.70.116:6123/user/jobmanager), starting network stack and library cache.
> 2018-04-29 08:24:32,987 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Determined BLOB server address to be /10.1.70.116:37931. Starting BLOB cache.
> 2018-04-29 08:24:32,987 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Created BLOB cache storage directory /tmp/blobStore-14e1221d-bb62-4e38-8c5a-4b19d003498b
> 2018-04-29 08:24:33,010 INFO org.apache.flink.runtime.blob.TransientBlobCache - Created BLOB cache storage directory /tmp/blobStore-69deafd1-b90e-4c44-acc5-282bf9690f00
> 2018-04-29 11:32:20,287 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Disassociated]
> 2018-04-29 11:32:28,030 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:32:28,031 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:32:38,030 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:32:38,031 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:32:48,031 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:32:48,031 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:32:51,117 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - TaskManager akka://flink/user/taskmanager disconnects from JobManager akka.tcp://flink@10.1.70.116:6123/user/jobmanager: Old JobManager lost its leadership.
> 2018-04-29 11:32:51,117 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManager - Disassociating from JobManager
> 2018-04-29 11:32:51,118 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Shutting down BLOB cache
> 2018-04-29 11:32:51,118 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
> 2018-04-29 11:32:53,050 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:32:53,051 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:32:58,069 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:32:58,069 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:03,090 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:03,091 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:08,109 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:08,110 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:13,132 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:13,132 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:18,149 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:18,150 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:23,170 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:23,170 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:28,189 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:28,190 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:33,210 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:33,210 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:38,230 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:38,230 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:43,249 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:43,250 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:48,269 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:48,272 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:53,290 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:53,291 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:33:58,311 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:33:58,312 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:03,329 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:03,330 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:08,352 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:08,352 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:13,373 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:13,377 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:18,399 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:18,400 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:23,420 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:23,421 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:28,440 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:28,440 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:33,459 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:33,459 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:38,481 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:38,481 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:43,499 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:43,500 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:48,519 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:48,519 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:53,539 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:53,540 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:34:58,566 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:34:58,567 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:03,591 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:03,591 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:08,610 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:08,610 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:13,629 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:13,630 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:18,650 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:18,651 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:23,672 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:23,673 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:28,689 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:28,690 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:33,709 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:33,716 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:38,729 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:38,731 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:43,750 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:43,751 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:48,769 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [null] failed with java.net.ConnectException: Connection refused: /10.1.70.116:6123
> 2018-04-29 11:35:48,770 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@10.1.70.116:6123] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://flink@10.1.70.116:6123]] Caused by: [Connection refused: /10.1.70.116:6123]
> 2018-04-29 11:35:53,787 ERROR akka.remote.Remoting - Association to [akka.tcp://flink@10.1.70.116:6123] with UID [-853636033] irrecoverably failed. Quarantining address.
> java.util.concurrent.TimeoutException: Delivery of system messages timed out and they were dropped.
> at akka.remote.ReliableDeliverySupervisor$$anonfun$gated$1.applyOrElse(Endpoint.scala:346)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
> at akka.remote.ReliableDeliverySupervisor.aroundReceive(Endpoint.scala:203)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
> at akka.actor.ActorCell.invoke(ActorCell.scala:495)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
> at akka.dispatch.Mailbox.run(Mailbox.scala:224)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 2018-04-29 11:42:20,322 INFO org.apache.flink.mesos.runtime.clusterframework.MesosTaskManagerRunner - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
> 2018-04-29 11:42:20,337 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager removed spill file directory /tmp/flink-io-bf946b07-190d-4e16-a5dc-fa76bc8498db
>
>
>