You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by 赵一旦 <hi...@gmail.com> on 2020/08/19 07:11:11 UTC
Flink运行偶尔会报AskTimeoutException等
如题,下面给出了一个异常栈。
有时候提交任务也会导致flink-webui类似卡死一样大概几十秒。任务包50MB左右,提交到远程集群,提交毕竟慢,需要几十秒,快1min提交。
13:10:54.136 [flink-scheduler-1] ERROR
org.apache.flink.runtime.rest.handler.job.JobDetailsHandler - Unhandled
exception.akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://
flink@cq02-frominf-temp25.cq02.xxx.com:2000/user/dispatcher#-1139504499]]
after [10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at
akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
at
scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109) at
scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
at
java.lang.Thread.run(Thread.java:748)13:10:54.140 [flink-scheduler-1] ERROR
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler
- Unhandled exception.akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://
flink@cq02-frominf-temp25.cq02.xxx.com:2000/user/dispatcher#-1139504499]]
after [10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at
akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
at
scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109) at
scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
at
java.lang.Thread.run(Thread.java:748)13:10:54.141 [flink-scheduler-1] ERROR
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointConfigHandler
- Unhandled exception.akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://
flink@cq02-frominf-temp25.cq02.xxx.com:2000/user/dispatcher#-1139504499]]
after [10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at
akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
at
scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109) at
scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
at
java.lang.Thread.run(Thread.java:748)13:11:04.225 [flink-scheduler-1] ERROR
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointConfigHandler
- Unhandled exception.akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://
flink@cq02-frominf-temp25.cq02.xxx.com:2000/user/dispatcher#-1139504499]]
after [10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at
akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
at
scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109) at
scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
at
java.lang.Thread.run(Thread.java:748)13:11:04.226 [flink-scheduler-1] ERROR
org.apache.flink.runtime.rest.handler.job.JobDetailsHandler - Unhandled
exception.akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://
flink@cq02-frominf-temp25.cq02.xxx.com:2000/user/dispatcher#-1139504499]]
after [10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at
akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
at
scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109) at
scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
at
java.lang.Thread.run(Thread.java:748)13:11:04.226 [flink-scheduler-1] ERROR
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler
- Unhandled exception.akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://
flink@cq02-frominf-temp25.cq02.xxx.com:2000/user/dispatcher#-1139504499]]
after [10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at
akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
at
scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109) at
scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
at
java.lang.Thread.run(Thread.java:748)13:11:07.205 [flink-scheduler-1] ERROR
org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler - Unhandled
exception.akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka.tcp://
flink@cq02-frominf-temp25.cq02.xxx.com:2000/user/dispatcher#-1139504499]]
after [10000 ms]. Message of type
[org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at
akka.pattern.PromiseActorRef$.$anonfun$defaultOnTimeout$1(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$.$anonfun$apply$1(AskSupport.scala:650) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:870)
at
scala.concurrent.BatchingExecutor.execute(BatchingExecutor.scala:109) at
scala.concurrent.BatchingExecutor.execute$(BatchingExecutor.scala:103) at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:868)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
at
java.lang.Thread.run(Thread.java:748)