You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2021/10/13 13:50:17 UTC
[GitHub] [dubbo] zhangyz-hd opened a new issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
zhangyz-hd opened a new issue #8985:
URL: https://github.com/apache/dubbo/issues/8985
- [x] I have searched the [issues](https://github.com/apache/dubbo/issues) of this repository and believe that this is not a duplicate.
- [x] I have checked the [FAQ](https://github.com/apache/dubbo/blob/master/FAQ.md) of this repository and believe that this is not a duplicate.
### Environment
* Dubbo version: 3.0.3/3.0.4-SNAPSHOT
* Operating System version: ANY
* Java version: 8
### Steps to reproduce this issue
1. 以最简单的demo配置启动服务提供者
```
dubbo.application.name=dubbo-demo-annotation-provider
dubbo.protocol.name=dubbo
dubbo.protocol.port=20881
dubbo.application.metadataType=remote
dubbo.registry.address=zookeeper://127.0.0.1:2181
dubbo.metadata-report.address=zookeeper://127.0.0.1:2182
```
2. 启动后，等待一会儿，退出java进程，出现AbstractServiceDiscovery.update相关的异常日志
```
org.apache.dubbo.rpc.RpcException: Failed register instance DefaultServiceInstance{, serviceName='dubbo-demo-annotation-provider', host='192.168.3.50', port=20881, enabled=true, healthy=true, metadata={dubbo.metadata-service.url-params={"version":"1.0.0","dubbo":"2.0.2","release":"3.0.4-SNAPSHOT","port":"20881","protocol":"dubbo"}, dubbo.endpoints=[{"port":20881,"protocol":"dubbo"}], dubbo.metadata.revision=0, dubbo.metadata.storage-type=remote}}
at org.apache.dubbo.registry.zookeeper.ZookeeperServiceDiscovery.doRegister(ZookeeperServiceDiscovery.java:97)
at org.apache.dubbo.registry.client.AbstractServiceDiscovery.register(AbstractServiceDiscovery.java:48)
at org.apache.dubbo.registry.zookeeper.ZookeeperServiceDiscovery.doUpdate(ZookeeperServiceDiscovery.java:105)
at org.apache.dubbo.registry.client.AbstractServiceDiscovery.update(AbstractServiceDiscovery.java:64)
at org.apache.dubbo.registry.client.metadata.ServiceInstanceMetadataUtils.lambda$refreshMetadataAndInstance$2(ServiceInstanceMetadataUtils.java:280)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.dubbo.registry.client.metadata.ServiceInstanceMetadataUtils.refreshMetadataAndInstance(ServiceInstanceMetadataUtils.java:270)
at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.lambda$registerServiceInstance$2(DefaultApplicationDeployer.java:714)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
或
```
[07/10/21 12:24:30:108 CST] Dubbo-framework-scheduler-thread-1 ERROR deploy.DefaultApplicationDeployer: [DUBBO] Refresh instance and metadata error, dubbo version: 3.0.4-SNAPSHOT, current host: 192.168.3.50
java.lang.RuntimeException
at org.apache.dubbo.common.function.ThrowableConsumer.execute(ThrowableConsumer.java:51)
at org.apache.dubbo.common.function.ThrowableConsumer.execute(ThrowableConsumer.java:64)
at org.apache.dubbo.registry.zookeeper.ZookeeperServiceDiscovery.doInServiceRegistry(ZookeeperServiceDiscovery.java:184)
at org.apache.dubbo.registry.zookeeper.ZookeeperServiceDiscovery.doUnregister(ZookeeperServiceDiscovery.java:110)
at org.apache.dubbo.registry.client.AbstractServiceDiscovery.unregister(AbstractServiceDiscovery.java:73)
at org.apache.dubbo.registry.zookeeper.ZookeeperServiceDiscovery.doUpdate(ZookeeperServiceDiscovery.java:104)
at org.apache.dubbo.registry.client.AbstractServiceDiscovery.update(AbstractServiceDiscovery.java:64)
at org.apache.dubbo.registry.client.metadata.ServiceInstanceMetadataUtils.lambda$refreshMetadataAndInstance$2(ServiceInstanceMetadataUtils.java:280)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.apache.dubbo.registry.client.metadata.ServiceInstanceMetadataUtils.refreshMetadataAndInstance(ServiceInstanceMetadataUtils.java:270)
at org.apache.dubbo.config.deploy.DefaultApplicationDeployer.lambda$registerServiceInstance$2(DefaultApplicationDeployer.java:714)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
### 简单分析
1. jvm退出，触发ServiceConfig#unexport，在InMemoryWritableMetadataService#unexportURL中释放了metadataSemaphore
2. 在DefaultApplicationDeployer#registerServiceInstance中，localMetadataService.blockUntilUpdated();获得metadataSemaphore，继续运行后续的ServiceInstanceMetadataUtils.refreshMetadataAndInstance。而此时jvm正在退出，部分对象已经销毁，无法完成一个完整的refreshMetadataAndInstance动作。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zhangyz-hd commented on issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
zhangyz-hd commented on issue #8985:
URL: https://github.com/apache/dubbo/issues/8985#issuecomment-997995643
Fixed in #9033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zhangyz-hd commented on issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
zhangyz-hd commented on issue #8985:
URL: https://github.com/apache/dubbo/issues/8985#issuecomment-942329117
#9002 并没有实际效果，这个我来跟进吧。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] kylixs commented on issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
kylixs commented on issue #8985:
URL: https://github.com/apache/dubbo/issues/8985#issuecomment-942943406
Some work is done in https://github.com/apache/dubbo/pull/9033, pls check it.
1. The application destroy processing is changed: https://github.com/apache/dubbo/pull/9033#discussion_r728613024
2. prevent refresh metadata in application stopping.
```java
if (registered) {
// scheduled task for updating Metadata and ServiceInstance
asyncMetadataFuture = executorRepository.getSharedScheduledExecutor().scheduleAtFixedRate(() -> {
// ignore refresh metadata on stopping
ApplicationDeployer deployer = applicationModel.getDeployer();
if (deployer != null && (deployer.isStopping() || deployer.isStopped())) {
return;
}
InMemoryWritableMetadataService localMetadataService = (InMemoryWritableMetadataService) WritableMetadataService.getDefaultExtension(applicationModel);
localMetadataService.blockUntilUpdated();
try {
ServiceInstanceMetadataUtils.refreshMetadataAndInstance(serviceInstance);
} catch (Exception e) {
logger.error("Refresh instance and metadata error", e);
} finally {
localMetadataService.releaseBlock();
}
}, 0, ConfigurationUtils.get(applicationModel, METADATA_PUBLISH_DELAY_KEY, DEFAULT_METADATA_PUBLISH_DELAY), TimeUnit.MILLISECONDS);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] wangchengming666 commented on issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
wangchengming666 commented on issue #8985:
URL: https://github.com/apache/dubbo/issues/8985#issuecomment-939254108
I'll fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ closed issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
AlbumenJ closed issue #8985:
URL: https://github.com/apache/dubbo/issues/8985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] kylixs edited a comment on issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
kylixs edited a comment on issue #8985:
URL: https://github.com/apache/dubbo/issues/8985#issuecomment-942943406
Some work is done in https://github.com/apache/dubbo/pull/9033, pls check it.
1. The application destroy processing is changed: https://github.com/apache/dubbo/pull/9033#discussion_r728613024
2. prevent refresh metadata in application stopping.
```java
if (registered) {
// scheduled task for updating Metadata and ServiceInstance
asyncMetadataFuture = executorRepository.getSharedScheduledExecutor().scheduleAtFixedRate(() -> {
// ignore refresh metadata on stopping
if (applicationModel.isDestroyed()) {
return;
}
InMemoryWritableMetadataService localMetadataService = (InMemoryWritableMetadataService) WritableMetadataService.getDefaultExtension(applicationModel);
localMetadataService.blockUntilUpdated();
try {
if (!applicationModel.isDestroyed()) {
ServiceInstanceMetadataUtils.refreshMetadataAndInstance(serviceInstance);
}
} catch (Exception e) {
if (!applicationModel.isDestroyed()) {
logger.error("Refresh instance and metadata error", e);
}
} finally {
localMetadataService.releaseBlock();
}
}, 0, ConfigurationUtils.get(applicationModel, METADATA_PUBLISH_DELAY_KEY, DEFAULT_METADATA_PUBLISH_DELAY), TimeUnit.MILLISECONDS);
}
```
```
public void blockUntilUpdated() {
try {
metadataSemaphore.tryAcquire(metadataPublishDelayTime, TimeUnit.MILLISECONDS);
metadataSemaphore.drainPermits();
updateLock.writeLock().lock();
} catch (InterruptedException e) {
if (!applicationModel.isDestroyed()) {
logger.warn("metadata refresh thread has been interrupted unexpectedly while waiting for update.", e);
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ closed issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
AlbumenJ closed issue #8985:
URL: https://github.com/apache/dubbo/issues/8985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zhangyz-hd commented on issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
zhangyz-hd commented on issue #8985:
URL: https://github.com/apache/dubbo/issues/8985#issuecomment-942329117
#9002 并没有实际效果，这个我来跟进吧。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] zhangyz-hd closed issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
zhangyz-hd closed issue #8985:
URL: https://github.com/apache/dubbo/issues/8985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] kylixs edited a comment on issue #8985: [3.0] RpcException/RuntimeException when Provider jvm exit
Posted by GitBox <gi...@apache.org>.
kylixs edited a comment on issue #8985:
URL: https://github.com/apache/dubbo/issues/8985#issuecomment-942943406
Some work is done in https://github.com/apache/dubbo/pull/9033, pls check it.
1. The application destroy processing is changed: https://github.com/apache/dubbo/pull/9033#discussion_r728613024
2. prevent refresh metadata in application stopping, ignore all relatived exceptions.
```java
if (registered) {
// scheduled task for updating Metadata and ServiceInstance
asyncMetadataFuture = executorRepository.getSharedScheduledExecutor().scheduleAtFixedRate(() -> {
// ignore refresh metadata on stopping
if (applicationModel.isDestroyed()) {
return;
}
InMemoryWritableMetadataService localMetadataService = (InMemoryWritableMetadataService) WritableMetadataService.getDefaultExtension(applicationModel);
localMetadataService.blockUntilUpdated();
try {
if (!applicationModel.isDestroyed()) {
ServiceInstanceMetadataUtils.refreshMetadataAndInstance(serviceInstance);
}
} catch (Exception e) {
if (!applicationModel.isDestroyed()) {
logger.error("Refresh instance and metadata error", e);
}
} finally {
localMetadataService.releaseBlock();
}
}, 0, ConfigurationUtils.get(applicationModel, METADATA_PUBLISH_DELAY_KEY, DEFAULT_METADATA_PUBLISH_DELAY), TimeUnit.MILLISECONDS);
}
```
```java
public void blockUntilUpdated() {
try {
metadataSemaphore.tryAcquire(metadataPublishDelayTime, TimeUnit.MILLISECONDS);
metadataSemaphore.drainPermits();
updateLock.writeLock().lock();
} catch (InterruptedException e) {
if (!applicationModel.isDestroyed()) {
logger.warn("metadata refresh thread has been interrupted unexpectedly while waiting for update.", e);
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org