You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@servicecomb.apache.org by GitBox <gi...@apache.org> on 2019/12/25 01:50:03 UTC
[GitHub] [servicecomb-java-chassis] Nick-The-Uncharted opened a new issue
#1489: Check whether current thread is eventLoopThread before calling other
microservice synchronously
Nick-The-Uncharted opened a new issue #1489: Check whether current thread is eventLoopThread before calling other microservice synchronously
URL: https://github.com/apache/servicecomb-java-chassis/issues/1489
In our microservice there were a chance that after calling a microservice, no packet was sent but `
vertx-blocked-thread-checker` continuously print following logs:
```
2019-12-24 13:36:56.097|[]|vertx-blocked-thread-checker|WARN |BlockedThreadChecker.java:55|Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 5389 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
at sun.misc.Unsafe.park(Native Method) ~[?:1.8.0_222]
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) ~[?:1.8.0_222]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838) ~[?:1.8.0_222]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:999) ~[?:1.8.0_222]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1306) ~[?:1.8.0_222]
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) ~[?:1.8.0_222]
at org.apache.servicecomb.core.provider.consumer.SyncResponseExecutor.waitResponse(SyncResponseExecutor.java:53) ~[java-chassis-core-1.3.0.jar!/:1.3.0]
at org.apache.servicecomb.core.provider.consumer.InvokerUtils.innerSyncInvoke(InvokerUtils.java:75) ~[java-chassis-core-1.3.0.jar!/:1.3.0]
at org.apache.servicecomb.provider.pojo.Invoker.syncInvoke(Invoker.java:174) ~[provider-pojo-1.3.0.jar!/:1.3.0]
at org.apache.servicecomb.provider.pojo.Invoker.invoke(Invoker.java:161) ~[provider-pojo-1.3.0.jar!/:1.3.0]
```
We finally found out its due to we are running our bussiniess logic in a async callback to other service which runs in eventLoopThread. When it calls another microservice synchronously，if vertx happens to pick the same eventLoopThread for connecting then there is deadlock.
Maybe check whether current thread is eventLoopThread before calling other microservice synchronously and print some warning log for troubleshooting such like problem is helpful.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-java-chassis] Nick-The-Uncharted commented on issue
#1489: [Suggestion]Check whether current thread is eventLoopThread before
calling other microservice synchronously
Posted by GitBox <gi...@apache.org>.
Nick-The-Uncharted commented on issue #1489: [Suggestion]Check whether current thread is eventLoopThread before calling other microservice synchronously
URL: https://github.com/apache/servicecomb-java-chassis/issues/1489#issuecomment-568945296
@liubao68 I think 99% of "blocking" case is calling other microservice, so its nice to print some warning log. Plus for the CompletableFuture return value scenario, developers wont even realize that they are using eventLoopThread to run their code. Maybe sepearate reactiveExecutor from vertxEventLoopExecutor is a good idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-java-chassis] Nick-The-Uncharted commented on issue
#1489: [Suggestion]Check whether current thread is eventLoopThread before
calling other microservice synchronously
Posted by GitBox <gi...@apache.org>.
Nick-The-Uncharted commented on issue #1489: [Suggestion]Check whether current thread is eventLoopThread before calling other microservice synchronously
URL: https://github.com/apache/servicecomb-java-chassis/issues/1489#issuecomment-568872827
Just ran into another scenario which may cause this: with return value as CompletableFuture and using handle to access the auth server.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-java-chassis] liubao68 commented on issue #1489:
[Suggestion]Check whether current thread is eventLoopThread before calling
other microservice synchronously
Posted by GitBox <gi...@apache.org>.
liubao68 commented on issue #1489: [Suggestion]Check whether current thread is eventLoopThread before calling other microservice synchronously
URL: https://github.com/apache/servicecomb-java-chassis/issues/1489#issuecomment-568942682
Application developers should keep the golden rule when writing reactive applications "do not execute blocking code in event-loop". The difficult for your suggestion is "How to check user code is blocking or not" and "Do not do this VS do it we make it work".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [servicecomb-java-chassis] Nick-The-Uncharted commented on issue
#1489: [Suggestion]Check whether current thread is eventLoopThread before
calling other microservice synchronously
Posted by GitBox <gi...@apache.org>.
Nick-The-Uncharted commented on issue #1489: [Suggestion]Check whether current thread is eventLoopThread before calling other microservice synchronously
URL: https://github.com/apache/servicecomb-java-chassis/issues/1489#issuecomment-568945458
And other blocking case is only slowing down the application. But calling other microservice is causing deadlock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services