You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "George Huang (Jira)" <ji...@apache.org> on 2022/01/06 03:51:00 UTC
[jira] [Commented] (HDDS-6156) OM node shutdown by ratis, multiple JVM pauses detected
[ https://issues.apache.org/jira/browse/HDDS-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17469660#comment-17469660 ]
George Huang commented on HDDS-6156:
------------------------------------
|11:03:04.998 PM|ERROR|GrpcUtil|Unexpected exception while waiting for channel termination
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelImpl.awaitTermination(ManagedChannelImpl.java:861)
at org.apache.ratis.thirdparty.io.grpc.internal.ForwardingManagedChannel.awaitTermination(ForwardingManagedChannel.java:57)
at org.apache.ratis.grpc.GrpcUtil.shutdownManagedChannel(GrpcUtil.java:215)
at org.apache.ratis.grpc.server.GrpcServerProtocolClient.close(GrpcServerProtocolClient.java:92)
at org.apache.ratis.util.PeerProxyMap.closeProxy(PeerProxyMap.java:166)
at org.apache.ratis.util.PeerProxyMap.lambda$null$3(PeerProxyMap.java:160)
at java.util.Optional.ifPresent(Optional.java:159)
at org.apache.ratis.util.PeerProxyMap.lambda$close$4(PeerProxyMap.java:160)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.util.concurrent.ConcurrentHashMap$ValueSpliterator.forEachRemaining(ConcurrentHashMap.java:3566)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:650)
at org.apache.ratis.util.PeerProxyMap.close(PeerProxyMap.java:159)
at org.apache.ratis.server.RaftServerRpcWithProxy.closeImpl(RaftServerRpcWithProxy.java:85)
at org.apache.ratis.grpc.server.GrpcService.closeImpl(GrpcService.java:264)
at org.apache.ratis.util.LifeCycle.lambda$checkStateAndClose$4(LifeCycle.java:299)
at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:319)
at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:297)
at org.apache.ratis.server.RaftServerRpcWithProxy.close(RaftServerRpcWithProxy.java:80)
at org.apache.ratis.server.impl.RaftServerProxy.lambda$close$6(RaftServerProxy.java:395)
at org.apache.ratis.util.LifeCycle.lambda$checkStateAndClose$4(LifeCycle.java:299)
at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:319)
at org.apache.ratis.util.LifeCycle.checkStateAndClose(LifeCycle.java:297)
at org.apache.ratis.server.impl.RaftServerProxy.close(RaftServerProxy.java:390)
at org.apache.ratis.server.impl.RaftServerProxy.handleJvmPause(RaftServerProxy.java:217)
at org.apache.ratis.server.impl.RaftServerProxy.lambda$new$0(RaftServerProxy.java:211)
at org.apache.ratis.util.JvmPauseMonitor.handle(JvmPauseMonitor.java:132)
at org.apache.ratis.util.JvmPauseMonitor.detectPause(JvmPauseMonitor.java:127)
at org.apache.ratis.util.JvmPauseMonitor.run(JvmPauseMonitor.java:106)
at java.lang.Thread.run(Thread.java:748)|
> OM node shutdown by ratis, multiple JVM pauses detected
> -------------------------------------------------------
>
> Key: HDDS-6156
> URL: https://issues.apache.org/jira/browse/HDDS-6156
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: George Huang
> Priority: Major
>
> OM node went down, some OM log:
> :
> |10:59:34.458 PM|WARN|FollowerState|Unexpected long sleep: sleep 5146ms but took extra 1237405308ns (> threshold = 300ms)|
> |10:59:34.488 PM|WARN|JvmPauseMonitor|JvmPauseMonitor-om3: Detected pause in JVM or host machine (eg GC): pause of approximately 2946723943ns.
> GC pool 'ParNew' had collection(s): count=1 time=3009ms|
> |10:59:34.488 PM|INFO|JvmPauseMonitor|Detected pause in JVM or host machine (eg GC): pause of approximately 2946ms
> GC pool 'ParNew' had collection(s): count=1 time=3009ms|
> |10:59:48.537 PM|INFO|JvmPauseMonitor|Detected pause in JVM or host machine (eg GC): pause of approximately 3046ms
> GC pool 'ParNew' had collection(s): count=1 time=3472ms|
> |10:59:48.537 PM|WARN|JvmPauseMonitor|JvmPauseMonitor-om3: Detected pause in JVM or host machine (eg GC): pause of approximately 3046802877ns.
> GC pool 'ParNew' had collection(s): count=1 time=3472ms|
> |11:00:01.829 PM|WARN|FollowerState|Unexpected long sleep: sleep 5170ms but took extra 1817775779ns (> threshold = 300ms)|
> |11:00:01.830 PM|WARN|JvmPauseMonitor|JvmPauseMonitor-om3: Detected pause in JVM or host machine (eg GC): pause of approximately 2791064147ns.
> GC pool 'ParNew' had collection(s): count=1 time=3116ms|
> :
> |11:00:11.189 PM|WARN|JvmPauseMonitor|JvmPauseMonitor-om3: Detected pause in JVM or host machine (eg GC): pause of approximately 1857446017ns. No GCs detected.|
> |11:03:04.457 PM|WARN|FollowerState|Unexpected long sleep: sleep 5049ms but took extra 167276471590ns (> threshold = 300ms)|
> |11:03:04.457 PM|WARN|JvmPauseMonitor|JvmPauseMonitor-om3: Detected pause in JVM or host machine (eg GC): pause of approximately 168767799439ns.
> GC pool 'ParNew' had collection(s): count=1 time=0ms
> GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=169150ms|
> |11:03:04.458 PM|WARN|JvmPauseMonitor|Detected pause in JVM or host machine (eg GC): pause of approximately 168767ms
> GC pool 'ParNew' had collection(s): count=1 time=0ms
> GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=169150ms|
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org