You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2021/09/15 05:17:35 UTC
[GitHub] [dubbo] zrlw opened a new issue #8805: [3.0] 集成构建，DubboShutdownHook线程频繁抛IllegalStateException异常：Shutdown in progress
zrlw opened a new issue #8805:
URL: https://github.com/apache/dubbo/issues/8805
### Environment
* Dubbo version: 3.0
github集成构建DubboShutdownHook线程抛IllegalStateException: Shutdown in progress，并且近期出现的频率越来越频繁：
```
2021-09-15T03:06:39.5680207Z ##[error]Exception in thread "DubboShutdownHook" java.lang.RuntimeException: java.lang.IllegalStateException: Shutdown in progress
2021-09-15T03:06:39.5694633Z at org.apache.dubbo.common.function.ThrowableAction.execute(ThrowableAction.java:48)
2021-09-15T03:06:39.5697163Z at org.apache.dubbo.common.lang.ShutdownHookCallbacks.lambda$callback$0(ShutdownHookCallbacks.java:72)
2021-09-15T03:06:39.5699110Z at java.lang.Iterable.forEach(Iterable.java:75)
2021-09-15T03:06:39.5700893Z at org.apache.dubbo.common.lang.ShutdownHookCallbacks.callback(ShutdownHookCallbacks.java:72)
2021-09-15T03:06:39.5703192Z at org.apache.dubbo.config.DubboShutdownHook.callback(DubboShutdownHook.java:89)
2021-09-15T03:06:39.5705079Z at org.apache.dubbo.config.DubboShutdownHook.run(DubboShutdownHook.java:76)
2021-09-15T03:06:39.5706254Z Caused by: java.lang.IllegalStateException: Shutdown in progress
2021-09-15T03:06:39.5707469Z at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
2021-09-15T03:06:39.5709546Z at java.lang.Runtime.removeShutdownHook(Runtime.java:239)
2021-09-15T03:06:39.5711564Z at org.apache.dubbo.config.DubboShutdownHook.unregister(DubboShutdownHook.java:115)
2021-09-15T03:06:39.5714856Z at org.apache.dubbo.config.bootstrap.DefaultApplicationDeployer.unRegisterShutdownHook(DefaultApplicationDeployer.java:162)
2021-09-15T03:06:39.5718071Z at org.apache.dubbo.config.bootstrap.DefaultApplicationDeployer.destroy(DefaultApplicationDeployer.java:849)
2021-09-15T03:06:39.5720445Z at org.apache.dubbo.common.function.ThrowableAction.execute(ThrowableAction.java:46)
```
看了代码是ApplicationShutdownHooks的remove方法判断hooks集合空所致：
```
if(hooks == null)
throw new IllegalStateException("Shutdown in progress");
```
问题：
1. DubboShutdownHook代码搞的是多实例，但类上的注解还在说单例：
```
This is a singleton in order to ensure there is only one shutdown hook registered. Because {@link ApplicationShutdownHooks} use {@link java.util.IdentityHashMap} to store the shutdown hooks.
```
2. 日志内容没有完整堆栈信息，所以不知道什么触发的org.apache.dubbo.common.function.ThrowableAction.execute，以及为什么此时ApplicationShutdownHooks的hooks已经为空。
在ShutdownHookCallbacks的callback()的循环里加try-catch可以解决构建失败问题：
```
private static final Logger logger = LoggerFactory.getLogger(ShutdownHookCallbacks.class);
public void callback() {
getCallbacks().forEach(callback -> {
try {
execute(callback::callback);
} catch (Exception e) {
logger.warn("callback exception.", e);
}
});
}
```
只是现在构建会频繁出现这个问题的原因是什么？
PS：
master分支的上述代码和3.0基本相同，不清楚是否存在类似的问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw edited a comment on issue #8805: [3.0] 近期集成构建结束前，DubboShutdownHook线程经常抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw edited a comment on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-919741212
更新：ReferenceCountExchangeClient的replaceWithLazyClient问题另外开issue了。
```
2021-09-15T03:06:39.5666317Z [15/09/21 03:06:36:036 UTC] DubboShutdownHook WARN dubbo.ReferenceCountExchangeClient: [DUBBO] 192.168.16.2:20880 org.apache.dubbo.samples.annotation.api.HelloService:1.0.0_annotation safe guard client , should not be called ,must have a bug., dubbo version: 3.0.3-SNAPSHOT, current host: 192.168.16.3
```
搜了一下safe guard client , should not be called ,must have a bug的代码有两处，一处是LazyConnectExchangeClient的warning
私有方法，由LazyConnectExchangeClient的request方法调起（感觉不应该是这里）：
warning方法注解说warn once every 5000 invocations，但是第一次调用就会warning，因为0模多少都还是0：
```
if (warningcount.get() % warning_period == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
warningcount.incrementAndGet();
```
另一处是ReferenceCountExchangeClient的replaceWithLazyClient，这个代码由close方法触发，这一处是日志告警的位置(日志有显示类名: WARN dubbo.ReferenceCountExchangeClient)。
```
if (disconnectCount.getAndIncrement() % maxDisconnectCount == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
```
此处和LazyConnectExchangeClient的warning方法一样，也存在第一次就warning的问题；此外replaceWithLazyClient方法的lazyUrl赋值语句靠前了，挪到后面可能合适一点：
```
if (!(client instanceof LazyConnectExchangeClient) || client.isClosed()) {
URL lazyUrl = url.addParameter(...); <=== 搁到这里就好
client = new LazyConnectExchangeClient(lazyUrl, client.getExchangeHandler());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw edited a comment on issue #8805: [3.0] 集成构建，DubboShutdownHook线程频繁抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw edited a comment on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-919741212
前面还有
```
2021-09-15T03:06:39.5666317Z [15/09/21 03:06:36:036 UTC] DubboShutdownHook WARN dubbo.ReferenceCountExchangeClient: [DUBBO] 192.168.16.2:20880 org.apache.dubbo.samples.annotation.api.HelloService:1.0.0_annotation safe guard client , should not be called ,must have a bug., dubbo version: 3.0.3-SNAPSHOT, current host: 192.168.16.3
```
搜了一下safe guard client , should not be called ,must have a bug的代码有两处，一处是LazyConnectExchangeClient的warning
私有方法，由LazyConnectExchangeClient的request方法调起（感觉不应该是这里）：
warning方法注解说warn once every 5000 invocations，但是第一次调用就会warning，因为0模多少都还是0：
```
if (warningcount.get() % warning_period == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
warningcount.incrementAndGet();
```
另一处是ReferenceCountExchangeClient的replaceWithLazyClient，这个代码由close方法触发，这一处是日志告警的位置(日志有显示类名: WARN dubbo.ReferenceCountExchangeClient)。
```
if (disconnectCount.getAndIncrement() % maxDisconnectCount == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
```
此处和LazyConnectExchangeClient的warning方法一样，也存在第一次就warning的问题；此外replaceWithLazyClient方法的lazyUrl赋值语句靠前了，挪到后面可能合适一点：
```
if (!(client instanceof LazyConnectExchangeClient) || client.isClosed()) {
URL lazyUrl = url.addParameter(...); <=== 搁到这里就好
client = new LazyConnectExchangeClient(lazyUrl, client.getExchangeHandler());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw edited a comment on issue #8805: [3.0] 集成构建，DubboShutdownHook线程频繁抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw edited a comment on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-919741212
前面还有
```
2021-09-15T03:06:39.5666317Z [15/09/21 03:06:36:036 UTC] DubboShutdownHook WARN dubbo.ReferenceCountExchangeClient: [DUBBO] 192.168.16.2:20880 org.apache.dubbo.samples.annotation.api.HelloService:1.0.0_annotation safe guard client , should not be called ,must have a bug., dubbo version: 3.0.3-SNAPSHOT, current host: 192.168.16.3
```
搜了一下safe guard client , should not be called ,must have a bug的代码有两处，一处是LazyConnectExchangeClient的warning
私有方法，由LazyConnectExchangeClient的request方法调起（感觉不应该是这里）：
warning方法注解说warn once every 5000 invocations，但是第一次调用就会warning，因为0模多少都还是0：
```
if (warningcount.get() % warning_period == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
warningcount.incrementAndGet();
```
另一处是ReferenceCountExchangeClient的replaceWithLazyClient，这个代码由close方法触发，这一处是日志告警的位置。
```
if (disconnectCount.getAndIncrement() % maxDisconnectCount == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
```
此处和LazyConnectExchangeClient的warning方法一样，也存在第一次就warning的问题；此外replaceWithLazyClient方法的lazyUrl赋值语句靠前了，挪到后面可能合适一点：
```
if (!(client instanceof LazyConnectExchangeClient) || client.isClosed()) {
URL lazyUrl = url.addParameter(...); <=== 搁到这里就好
client = new LazyConnectExchangeClient(lazyUrl, client.getExchangeHandler());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw commented on issue #8805: [3.0] 近期集成构建结束前，DubboShutdownHook线程经常抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw commented on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-940961574
可能是测试类结束时没有关闭zkClient所致，这种情况下，zk的ClientCnxn的内部类SendThread的run方法会不停地调用clientCnxnSocket.doTransport重连已经关闭的zk Server，但是每次重连都因为zk Server已关闭都失败，这种设计不是脑残，而是zk的自动重连机制所致，只要不close，它会一直死循环下去，直到应用程序退出为止。
#8993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw edited a comment on issue #8805: [3.0] 集成构建，DubboShutdownHook线程频繁抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw edited a comment on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-919741212
前面还有
```
2021-09-15T03:06:39.5666317Z [15/09/21 03:06:36:036 UTC] DubboShutdownHook WARN dubbo.ReferenceCountExchangeClient: [DUBBO] 192.168.16.2:20880 org.apache.dubbo.samples.annotation.api.HelloService:1.0.0_annotation safe guard client , should not be called ,must have a bug., dubbo version: 3.0.3-SNAPSHOT, current host: 192.168.16.3
```
搜了一下safe guard client , should not be called ,must have a bug的代码有两处，一处是LazyConnectExchangeClient的warning
私有方法，由LazyConnectExchangeClient的request方法调起（感觉不应该是这里）：
warning方法注解说warn once every 5000 invocations，但是第一次调用就会warning，因为0模多少都还是0：
```
if (warningcount.get() % warning_period == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
warningcount.incrementAndGet();
```
另一处是ReferenceCountExchangeClient的replaceWithLazyClient，这个代码由close方法触发，这一处可能是日志告警的位置。
此处和LazyConnectExchangeClient的warning方法一样，也存在第一次就warning的问题；此外replaceWithLazyClient方法的lazyUrl赋值语句靠前了，挪到后面可能合适一点：
```
if (!(client instanceof LazyConnectExchangeClient) || client.isClosed()) {
URL lazyUrl = url.addParameter(...); <=== 搁到这里就好
client = new LazyConnectExchangeClient(lazyUrl, client.getExchangeHandler());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw commented on issue #8805: [3.0] 集成构建，DubboShutdownHook线程频繁抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw commented on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-919741212
前面还有LazyConnectExchangeClient的warning
```
2021-09-15T03:06:39.5666317Z [15/09/21 03:06:36:036 UTC] DubboShutdownHook WARN dubbo.ReferenceCountExchangeClient: [DUBBO] 192.168.16.2:20880 org.apache.dubbo.samples.annotation.api.HelloService:1.0.0_annotation safe guard client , should not be called ,must have a bug., dubbo version: 3.0.3-SNAPSHOT, current host: 192.168.16.3
```
warning方法的注解说warn once every 5000 invocations，但是第一次调用就会warning，因为warningcount为0，模多少都还是0：
```
if (warningcount.get() % warning_period == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
warningcount.incrementAndGet();
```
LazyConnectExchangeClient的request为什么会被DubboShutdownHook调用？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw edited a comment on issue #8805: [3.0] 集成构建，DubboShutdownHook线程频繁抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw edited a comment on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-919741212
前面还有
```
2021-09-15T03:06:39.5666317Z [15/09/21 03:06:36:036 UTC] DubboShutdownHook WARN dubbo.ReferenceCountExchangeClient: [DUBBO] 192.168.16.2:20880 org.apache.dubbo.samples.annotation.api.HelloService:1.0.0_annotation safe guard client , should not be called ,must have a bug., dubbo version: 3.0.3-SNAPSHOT, current host: 192.168.16.3
```
搜了一下safe guard client , should not be called ,must have a bug的代码有两处，一处是LazyConnectExchangeClient的warning
私有方法，由LazyConnectExchangeClient的request方法调起（感觉不应该是这里）：
warning方法注解说warn once every 5000 invocations，但是第一次调用就会warning，因为0模多少都还是0：
```
if (warningcount.get() % warning_period == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
warningcount.incrementAndGet();
```
另一处是ReferenceCountExchangeClient的replaceWithLazyClient，这个代码由close方法触发，这一处可能是日志告警的位置。
此处和LazyConnectExchangeClient的warning方法一样，也存在第一次就warning的问题；此外replaceWithLazyClient方法的lazyUrl赋值语句靠前了，挪到后面可能合适一点：
```
if (!(client instanceof LazyConnectExchangeClient) || client.isClosed()) {
URL lazyUrl = url.addParameter(...); <=== 搁到这里就好
client = new LazyConnectExchangeClient(lazyUrl, client.getExchangeHandler());
}
```
如果是replaceWithLazyClient方法输出的日志，那么说明ReferenceCountExchangeClient被close了多次。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw edited a comment on issue #8805: [3.0] 集成构建，DubboShutdownHook线程频繁抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw edited a comment on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-919741212
前面还有
```
2021-09-15T03:06:39.5666317Z [15/09/21 03:06:36:036 UTC] DubboShutdownHook WARN dubbo.ReferenceCountExchangeClient: [DUBBO] 192.168.16.2:20880 org.apache.dubbo.samples.annotation.api.HelloService:1.0.0_annotation safe guard client , should not be called ,must have a bug., dubbo version: 3.0.3-SNAPSHOT, current host: 192.168.16.3
```
搜了一下safe guard client , should not be called ,must have a bug的代码有两处，一处是LazyConnectExchangeClient的warning
私有方法，由LazyConnectExchangeClient的request方法调起（感觉不应该是这里）：
warning方法注解说warn once every 5000 invocations，但是第一次调用就会warning，因为0模多少都还是0：
```
if (warningcount.get() % warning_period == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
warningcount.incrementAndGet();
```
另一处是ReferenceCountExchangeClient的replaceWithLazyClient，这个代码由close方法触发，这一处可能是日志告警的位置。
```
if (disconnectCount.getAndIncrement() % maxDisconnectCount == 0) {
logger.warn(url.getAddress() + " " + url.getServiceKey() + " safe guard client , should not be called ,must have a bug.");
}
```
此处和LazyConnectExchangeClient的warning方法一样，也存在第一次就warning的问题；此外replaceWithLazyClient方法的lazyUrl赋值语句靠前了，挪到后面可能合适一点：
```
if (!(client instanceof LazyConnectExchangeClient) || client.isClosed()) {
URL lazyUrl = url.addParameter(...); <=== 搁到这里就好
client = new LazyConnectExchangeClient(lazyUrl, client.getExchangeHandler());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] shihzenq commented on issue #8805: [3.0] 近期集成构建结束前，DubboShutdownHook线程经常抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
shihzenq commented on issue #8805:
URL: https://github.com/apache/dubbo/issues/8805#issuecomment-939269371
大佬们，那应该怎么解决呢？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zrlw closed issue #8805: [3.0] 近期集成构建结束前，DubboShutdownHook线程经常抛IllegalStateException异常：Shutdown in progress
Posted by GitBox <gi...@apache.org>.
zrlw closed issue #8805:
URL: https://github.com/apache/dubbo/issues/8805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org