You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by crazy <24...@qq.com.INVALID> on 2023/04/23 06:28:01 UTC
flink rocksdb异常
Hi, 大佬好,
有个Flink on Yarn程序，Flink版本使用的是flink-1.13.5，statebackend使用的是rocksdb，任务跑一段时间，就会出现如下堆栈异常：
2023-04-20 22:32:08,127 INFO org.apache.flink.runtime.taskmanager.Task [] - Attempting to fail task externally Source: slow_source_name_3 (15/100)#0 (39e2f191bef3484c9b629258eb5afb87).
2023-04-20 22:32:08,146 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: slow_source_name_3 (15/100)#0 (39e2f191bef3484c9b629258eb5afb87) switched from RUNNING to FAILED with failure cause: java.util.concurrent.TimeoutException: Invocation of public abstract java.util.concurrent.CompletableFuture org.apache.flink.runtime.jobmaster.JobMasterOperatorEventGateway.sendOperatorEventToCoordinator(org.apache.flink.runtime.executiongraph.ExecutionAttemptID,org.apache.flink.runtime.jobgraph.OperatorID,org.apache.flink.util.SerializedValue) timed out.
at com.sun.proxy.$Proxy28.sendOperatorEventToCoordinator(Unknown Source)
at org.apache.flink.runtime.taskexecutor.rpc.RpcTaskOperatorEventGateway.sendOperatorEventToCoordinator(RpcTaskOperatorEventGateway.java:58)
at org.apache.flink.streaming.runtime.tasks.OperatorEventDispatcherImpl$OperatorEventGatewayImpl.sendEventToCoordinator(OperatorEventDispatcherImpl.java:114)
at org.apache.flink.streaming.api.operators.SourceOperator.emitLatestWatermark(SourceOperator.java:393)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1425)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1416)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
at org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.lang.Thread.run(Thread.java:748)
Caused by: akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://flink@xxxxxxxxxx:40381/user/rpc/jobmanager_2#-1762983006]] after [10000 ms]. Message of type [org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
... 1 more
查看对应进程日志，
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=551250, tid=0x00007fcb1e7fc700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C 0x0000000000000000
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /data1/hadoop/local/usercache/xxx/appcache/application_1681367620163_1307/container_e28_1681367620163_1307_01_000232/hs_err_pid551250.log
Compiled method (nm) 5494257 11953 n 0 org.rocksdb.RocksDB::iteratorCF (native)
total in heap [0x00007fcb8335de10,0x00007fcb8335e168] = 856
relocation [0x00007fcb8335df38,0x00007fcb8335df80] = 72
main code [0x00007fcb8335df80,0x00007fcb8335e160] = 480
oops [0x00007fcb8335e160,0x00007fcb8335e168] = 8
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
请问下这是什么原因呢?
crazy
2463829830@qq.com
Re: flink rocksdb异常
Posted by Shammon FY <zj...@gmail.com>.
Hi
这是TM向JM发送消息超时了，可以了看下JM是否有错误日志，或者对应的TM和JM是否有资源打满等情况，导致akka消息超时
Best,
Shammon FY
On Sun, Apr 23, 2023 at 2:28 PM crazy <24...@qq.com.invalid> wrote:
> Hi, 大佬好,
> 有个Flink on
> Yarn程序，Flink版本使用的是flink-1.13.5，statebackend使用的是rocksdb，任务跑一段时间，就会出现如下堆栈异常：
>
>
> 2023-04-20 22:32:08,127 INFO
> org.apache.flink.runtime.taskmanager.Task
> [] - Attempting to fail task externally
> Source: slow_source_name_3 (15/100)#0 (39e2f191bef3484c9b629258eb5afb87).
> 2023-04-20 22:32:08,146 WARN
> org.apache.flink.runtime.taskmanager.Task
> [] - Source: slow_source_name_3
> (15/100)#0 (39e2f191bef3484c9b629258eb5afb87) switched from RUNNING to
> FAILED with failure cause: java.util.concurrent.TimeoutException:
> Invocation of public abstract java.util.concurrent.CompletableFuture
> org.apache.flink.runtime.jobmaster.JobMasterOperatorEventGateway.sendOperatorEventToCoordinator(org.apache.flink.runtime.executiongraph.ExecutionAttemptID,org.apache.flink.runtime.jobgraph.OperatorID,org.apache.flink.util.SerializedValue)
> timed out.
>
> at
> com.sun.proxy.$Proxy28.sendOperatorEventToCoordinator(Unknown Source)
>
> at
> org.apache.flink.runtime.taskexecutor.rpc.RpcTaskOperatorEventGateway.sendOperatorEventToCoordinator(RpcTaskOperatorEventGateway.java:58)
>
> at
> org.apache.flink.streaming.runtime.tasks.OperatorEventDispatcherImpl$OperatorEventGatewayImpl.sendEventToCoordinator(OperatorEventDispatcherImpl.java:114)
>
> at
> org.apache.flink.streaming.api.operators.SourceOperator.emitLatestWatermark(SourceOperator.java:393)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1425)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$16(StreamTask.java:1416)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:344)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:330)
>
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:202)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:684)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.executeInvoke(StreamTask.java:639)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650)
>
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:623)
>
> at
> org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:779)
>
> at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
>
> at java.lang.Thread.run(Thread.java:748)
>
> Caused by: akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka.tcp://flink@xxxxxxxxxx:40381/user/rpc/jobmanager_2#-1762983006]]
> after [10000 ms]. Message of type
> [org.apache.flink.runtime.rpc.messages.RemoteFencedMessage]. A typical
> reason for `AskTimeoutException` is that the recipient actor didn't send a
> reply.
>
> at
> akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
>
> at
> akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
>
> at
> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
>
> at
> akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
>
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
>
> at
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
>
> at
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
>
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
>
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
>
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
>
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
>
> ... 1 more
>
>
>
>
>
> 查看对应进程日志，
>
> #
>
> # A fatal error has been detected by the Java Runtime Environment:
>
> #
>
> # SIGSEGV (0xb) at pc=0x0000000000000000, pid=551250,
> tid=0x00007fcb1e7fc700
>
> #
>
> # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build
> 1.8.0_131-b11)
>
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode
> linux-amd64 compressed oops)
>
> # Problematic frame:
>
> # C 0x0000000000000000
>
> #
>
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
>
> #
>
> # An error report file with more information is saved as:
>
> #
> /data1/hadoop/local/usercache/xxx/appcache/application_1681367620163_1307/container_e28_1681367620163_1307_01_000232/hs_err_pid551250.log
>
> Compiled method (nm) 5494257 11953 n 0
> org.rocksdb.RocksDB::iteratorCF (native)
>
> total in heap [0x00007fcb8335de10,0x00007fcb8335e168] = 856
>
> relocation [0x00007fcb8335df38,0x00007fcb8335df80] = 72
>
> main code
> [0x00007fcb8335df80,0x00007fcb8335e160] = 480
>
> oops
> [0x00007fcb8335e160,0x00007fcb8335e168] = 8
>
> #
>
> # If you would like to submit a bug report, please visit:
>
> # http://bugreport.java.com/bugreport/crash.jsp
>
> # The crash happened outside the Java Virtual Machine in native code.
>
> # See problematic frame for where to report the bug.
>
>
> 请问下这是什么原因呢?
>
>
> crazy
> 2463829830@qq.com
>
>
>
>