You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2022/08/16 11:27:07 UTC
[GitHub] [servicecomb-java-chassis] njhw12 opened a new issue, #3287: vertx-blocked-thread-checker出现超长时间阻塞
njhw12 opened a new issue, #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287
线程堆栈目前发现有两类：
1.
Thread Thread[transport-vert.x-eventloop-thread-1,5,main] has been blocked for 11900 ms, time limit is 2000 ms|io.vertx.core.impl.BlockedThreadChecker
io.vertx.core.VertxException: Thread blocked
at sun.misc.Unsafe.unpark(Native Method)
at java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1266)
at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
at java.util.concurrent.LinkedBlockingQueue.signalNotEmpty(LinkedBlockingQueue.java:176)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:430)
at org.apache.servicecomb.core.executor.LinkedBlockingQueueEx.offer(LinkedBlockingQueueEx.java:47)
at org.apache.servicecomb.core.executor.LinkedBlockingQueueEx.offer(LinkedBlockingQueueEx.java:22)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1371)
at org.apache.servicecomb.core.executor.ThreadPoolExecutorEx.execute(ThreadPoolExecutorEx.java:46)
at org.apache.servicecomb.core.executor.GroupExecutor.execute(GroupExecutor.java:141)
at org.apache.servicecomb.common.rest.AbstractRestInvocation.scheduleInvocation(AbstractRestInvocation.java:147)
at org.apache.servicecomb.common.rest.RestProducerInvocation.invoke(RestProducerInvocation.java:52)
at org.apache.servicecomb.transport.rest.vertx.VertxRestDispatcher.onRequest(VertxRestDispatcher.java:219)
at org.apache.servicecomb.transport.rest.vertx.VertxRestDispatcher$$Lambda$4251/1027787170.handle(Unknown Source)
at io.vertx.ext.web.impl.RouteState.handleContext(RouteState.java:1127)
at io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:151)
at io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)
at org.apache.servicecomb.transport.rest.vertx.RestBodyHandler$BHandler.doEnd(RestBodyHandler.java:362)
at org.apache.servicecomb.transport.rest.vertx.RestBodyHandler$BHandler.end(RestBodyHandler.java:339)
at org.apache.servicecomb.transport.rest.vertx.RestBodyHandler.lambda$handle$0(RestBodyHandler.java:112)
at org.apache.servicecomb.transport.rest.vertx.RestBodyHandler$$Lambda$4506/2073179658.handle(Unknown Source)
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:100)
at io.vertx.core.impl.AbstractContext.dispatch(AbstractContext.java:63)
at io.vertx.core.http.impl.HttpEventHandler.handleEnd(HttpEventHandler.java:76)
at io.vertx.core.http.impl.Http1xServerRequest.onEnd(Http1xServerRequest.java:565)
at io.vertx.core.http.impl.Http1xServerRequest.handleEnd(Http1xServerRequest.java:548)
at io.vertx.core.http.impl.Http1xServerConnection$$Lambda$4509/1354880134.handle(Unknown Source)
at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:71)
at io.vertx.core.impl.DuplicatedContext.execute(DuplicatedContext.java:163)
at io.vertx.core.http.impl.Http1xServerConnection.onEnd(Http1xServerConnection.java:189)
at io.vertx.core.http.impl.Http1xServerConnection.onContent(Http1xServerConnection.java:179)
at io.vertx.core.http.impl.Http1xServerConnection.handleOther(Http1xServerConnection.java:159)
at io.vertx.core.http.impl.Http1xServerConnection.handleMessage(Http1xServerConnection.java:147)
at io.vertx.core.net.impl.ConnectionBase.read(ConnectionBase.java:155)
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:154)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.vertx.core.http.impl.Http1xOrH2CHandler.end(Http1xOrH2CHandler.java:61)
at io.vertx.core.http.impl.Http1xOrH2CHandler.channelRead(Http1xOrH2CHandler.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
2.
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:817)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] vertx-blocked-thread-checker出现超长时间阻塞 [servicecomb-java-chassis]
Posted by "wyhluckydog (via GitHub)" <gi...@apache.org>.
wyhluckydog commented on issue #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287#issuecomment-1926617547
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] njhw12 commented on issue #3287: vertx-blocked-thread-checker出现超长时间阻塞
Posted by GitBox <gi...@apache.org>.
njhw12 commented on issue #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287#issuecomment-1219196720
>
这个不是队列吧，根据异常堆栈结合jdk的代码看是入队成功了，只是入队前队列为空，发送队列变为非空通知时，最后释放锁时阻塞了，而且如果怀疑是内存fullgc类的，也不该阻塞近12秒吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] vertx-blocked-thread-checker出现超长时间阻塞 [servicecomb-java-chassis]
Posted by "liubao68 (via GitHub)" <gi...@apache.org>.
liubao68 commented on issue #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287#issuecomment-1926608474
> I also encountered the same problem, the service is restored after restart, is there a solution？
Do you have the same stack-trace?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] vertx-blocked-thread-checker出现超长时间阻塞 [servicecomb-java-chassis]
Posted by "wyhluckydog (via GitHub)" <gi...@apache.org>.
wyhluckydog commented on issue #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287#issuecomment-1926618827
But I don't know how to deal with it. Disappointment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] vertx-blocked-thread-checker出现超长时间阻塞 [servicecomb-java-chassis]
Posted by "wyhluckydog (via GitHub)" <gi...@apache.org>.
wyhluckydog commented on issue #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287#issuecomment-1926590177
I also encountered the same problem, the service is restored after restart, is there a solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #3287: vertx-blocked-thread-checker出现超长时间阻塞
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287#issuecomment-1217791169
看起来是排队请求太多，队列都满了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [servicecomb-java-chassis] lvorytower commented on issue #3287: vertx-blocked-thread-checker出现超长时间阻塞
Posted by GitBox <gi...@apache.org>.
lvorytower commented on issue #3287:
URL: https://github.com/apache/servicecomb-java-chassis/issues/3287#issuecomment-1229134516
最后有结论吧吗？？
我也遇到了这个问题， 时延增加导致请求排队过多，队列阻塞。 阻塞了几百秒。
奇怪的是：后端时延恢复以后，还是继续阻塞。服务重启后，就立刻恢复了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@servicecomb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org