You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "chaoyoung (via GitHub)" <gi...@apache.org> on 2023/09/20 03:08:13 UTC
[GitHub] [dubbo] chaoyoung opened a new issue, #13081: ERROR: Directory of type RegistryDirectory already destroyed for service xxx while dubbo elegant shutdown
chaoyoung opened a new issue, #13081:
URL: https://github.com/apache/dubbo/issues/13081
### Environment
* Dubbo version: 3.2.6
* Operating System version: CentOS 7
* Java version: 1.8
### Steps to reproduce this issue
1. Consumer开启优雅下线：server.shutdown=graceful, spring.lifecycle.timeout-per-shutdown-phase=30s, dubbo.application.shutwait=30000
2. 停止Consumer的同时，不断地像Consumer的SpringMVC接口发送请求
3. 在Consumer的Web容器还没有完全关闭的时候，部分请求进来后由于无法向Provider发送请求而报错
### Expected Behavior
Dubbo消费者能在Spring Web容器关闭后再关闭，这样才能做到流量无损
### Actual Behavior
Dubbo消费者调用出错：Directory of type RegistryDirectory already destroyed for service xxx.
不知是否跟这个有关：`[DubboShutdownHook] org.apache.dubbo.config.DubboShutdownHook Line:130 - [DUBBO] Dubbo wait for application(Dubbo Application[1.1](crm-dubbo)) managed by Spring to be shutdown failed, time usage: 33ms`
```
2023-09-19 20:34:23.307 INFO [DubboShutdownHook] org.apache.dubbo.config.DubboShutdownHook Line:77 - [DUBBO] Run shutdown hook now., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:23.308 WARN [Thread-6] com.alibaba.nacos.common.notify.NotifyCenter Line:136 - [NotifyCenter] Start destroying Publisher
2023-09-19 20:34:23.309 WARN [Thread-6] com.alibaba.nacos.common.notify.NotifyCenter Line:153 - [NotifyCenter] Destruction of the end
2023-09-19 20:34:23.309 WARN [Thread-2] com.alibaba.nacos.common.http.HttpClientBeanHolder Line:102 - [HttpClientBeanHolder] Start destroying common HttpClient
2023-09-19 20:34:23.313 INFO [DubboShutdownHook] org.apache.dubbo.config.DubboShutdownHook Line:109 - [DUBBO] Waiting for modules(Dubbo Application[1.1](crm-dubbo)) managed by Spring to be shutdown., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:23.314 WARN [Thread-2] com.alibaba.nacos.common.http.HttpClientBeanHolder Line:111 - [HttpClientBeanHolder] Destruction of the end
2023-09-19 20:34:23.336 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultModuleDeployer Line:368 - [DUBBO] Dubbo Module[1.1.1] is stopping., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:23.342 INFO [SpringApplicationShutdownHook] org.apache.dubbo.registry.nacos.NacosRegistry Line:400 - [DUBBO] Unregister: tri://10.244.23.158:50051/com.example.cmc.console.api.service.IOperateResultService?application=crm-dubbo&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&interface=com.example.cmc.console.api.service.IOperateResultService&methods=saveOperateResult,selectOperateResultById,updateOperateResult&prefer.serialization=hessian2,fastjson2&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125300156&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:23.346 INFO [DubboShutdownHook] org.apache.dubbo.config.DubboShutdownHook Line:130 - [DUBBO] Dubbo wait for application(Dubbo Application[1.1](crm-dubbo)) managed by Spring to be shutdown failed, time usage: 33ms, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:23.346 INFO [DubboShutdownHook] org.apache.dubbo.config.DubboShutdownHook Line:136 - [DUBBO] Dubbo shutdown hooks execute now. Dubbo Application[1.1](crm-dubbo), dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:25.017 INFO [SpringApplicationShutdownHook] org.apache.dubbo.registry.nacos.NacosRegistry Line:430 - [DUBBO] Unsubscribe: consumer://10.244.23.158/com.example.cmc.user.api.service.ICompositeEmployeeService?application=crm-dubbo&background=false&category=providers,configurators,routers&check=false&check.serializable=true&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=com.example.cmc.user.api.service.ICompositeEmployeeService&methods=getEmployeeDepartment,getEmployeeDeptAndSubDeptIds,listEmployeeByIds,listEmployeeByRole,queryEmployeeRoleScope,queryEmployeeWithinDepartmentByKeyword&pid=1®ister-mode=all&release=3.2.6&retries=0&revision=1.1.0-SNAPSHOT&serialize.check.status=WARN&shutwait=30000&side=consumer&sticky=false×tamp=1695125308709&unloadClusterRelated=false&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:25.033 WARN [NettyServerWorker-13-2] org.apache.dubbo.remoting.transport.AbstractServer Line:? - [DUBBO] All clients has disconnected from 0.0.0.0/0.0.0.0:20880. You can graceful shutdown now., dubbo version: 3.2.6, current host: 10.244.23.158, error code: 99-0. This may be caused by unknown error in remoting module, go to https://dubbo.apache.org/faq/99/0 to find instructions.
2023-09-19 20:34:25.036 INFO [NettyServerWorker-13-2] org.apache.dubbo.remoting.transport.netty4.NettyServerHandler Line:90 - [DUBBO] The connection of /10.244.23.220:54952 -> 0.0.0.0/0.0.0.0:20880 is disconnected., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:25.996 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultModuleDeployer Line:378 - [DUBBO] Dubbo Module[1.1.1] has stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:25.996 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.model.FrameworkModel Line:308 - [DUBBO] Reset global default application from Dubbo Application[1.1](crm-dubbo) to null, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:25.996 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultApplicationDeployer Line:1242 - [DUBBO] Dubbo Application[1.1](crm-dubbo) is stopping., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:25.998 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol Line:543 - [DUBBO] Destroying protocol [DubboProtocol] ..., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:25.998 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.dubbo.DubboProtocol Line:556 - [DUBBO] Closing dubbo server: /10.244.23.158:20880, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:26.001 INFO [SpringApplicationShutdownHook] org.apache.dubbo.remoting.transport.AbstractServer Line:122 - [DUBBO] Close NettyServer bind /0.0.0.0:20880, export /10.244.23.158:20880, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:29.476 ERROR [XNIO-1 task-7] com.example.cmc.console.aop.GlobalExceptionHandler Line:177 - Directory of type RegistryDirectory already destroyed for service com.example.cmc.user.api.service.ICompositeEmployeeService:1.0 from registry nacos://nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?application=crm-dubbo&check=true&check.serializable=true&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&password=nacos&pid=1®ister=true®ister-mode=all&release=3.2.6&serialize.check.status=WARN&shutwait=30000&subscribe=true&timeout=5000×tamp=1695125308719&username=nacos
org.apache.dubbo.rpc.RpcException: Directory of type RegistryDirectory already destroyed for service com.example.cmc.user.api.service.ICompositeEmployeeService:1.0 from registry nacos://nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?application=crm-dubbo&check=true&check.serializable=true&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&password=nacos&pid=1®ister=true®ister-mode=all&release=3.2.6&serialize.check.status=WARN&shutwait=30000&subscribe=true&timeout=5000×tamp=1695125308719&username=nacos
at org.apache.dubbo.rpc.cluster.directory.AbstractDirectory.list(AbstractDirectory.java:194)
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.list(AbstractClusterInvoker.java:423)
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:334)
at org.apache.dubbo.rpc.cluster.router.RouterSnapshotFilter.invoke(RouterSnapshotFilter.java:46)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:108)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.rpc.cluster.filter.support.MetricsClusterFilter.invoke(MetricsClusterFilter.java:56)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:52)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.spring.security.filter.ContextHolderParametersSelectedTransferFilter.invoke(ContextHolderParametersSelectedTransferFilter.java:41)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.metrics.filter.MetricsFilter.invoke(MetricsFilter.java:73)
at org.apache.dubbo.rpc.cluster.filter.support.MetricsConsumerFilter.invoke(MetricsConsumerFilter.java:37)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerClassLoaderFilter.invoke(ConsumerClassLoaderFilter.java:40)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.rpc.cluster.filter.support.ConsumerContextFilter.invoke(ConsumerContextFilter.java:116)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CopyOfFilterChainNode.invoke(FilterChainBuilder.java:334)
at org.apache.dubbo.rpc.cluster.filter.FilterChainBuilder$CallbackRegistrationInvoker.invoke(FilterChainBuilder.java:196)
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$ClusterFilterInvoker.invoke(AbstractCluster.java:91)
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:103)
at org.apache.dubbo.rpc.cluster.support.wrapper.ScopeClusterInvoker.invoke(ScopeClusterInvoker.java:169)
at org.apache.dubbo.registry.client.migration.MigrationInvoker.invoke(MigrationInvoker.java:284)
at org.apache.dubbo.rpc.proxy.InvocationUtil.invoke(InvocationUtil.java:61)
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:75)
at com.example.cmc.user.api.service.ICompositeEmployeeServiceDubboProxy0.queryEmployeeRoleScope(ICompositeEmployeeServiceDubboProxy0.java)
at sun.reflect.GeneratedMethodAccessor769.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.dubbo.config.spring.util.LazyTargetInvocationHandler.invoke(LazyTargetInvocationHandler.java:58)
at com.example.cmc.user.api.service.ICompositeEmployeeServiceDubboProxy0.queryEmployeeRoleScope(ICompositeEmployeeServiceDubboProxy0.java)
at com.example.cmc.console.aop.PermissionAop.setPermissionQuery(PermissionAop.java:65)
at com.example.cmc.console.aop.PermissionAop.lambda$doBeforeAdvice$0(PermissionAop.java:43)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at com.example.cmc.console.aop.PermissionAop.doBeforeAdvice(PermissionAop.java:41)
at sun.reflect.GeneratedMethodAccessor499.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:617)
at org.springframework.aop.aspectj.AspectJMethodBeforeAdvice.before(AspectJMethodBeforeAdvice.java:44)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at com.example.cmc.console.controller.CallTaskController$$EnhancerBySpringCGLIB$$3cd9409c.listByTemplateCode(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at com.example.common.http.filter.AbstractHttpRequestLoggingFilter.doFilterInternal(AbstractHttpRequestLoggingFilter.java:228)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at com.example.common.http.filter.XssFilter.doFilter(XssFilter.java:22)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
at java.lang.Thread.run(Thread.java:748)
2023-09-19 20:34:30.051 INFO [SpringApplicationShutdownHook] org.apache.dubbo.qos.server.Server Line:133 - [DUBBO] qos-server stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.057 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.injvm.InjvmProtocol Line:124 - [DUBBO] Unexport service: injvm://127.0.0.1/org.apache.dubbo.metadata.MetadataService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=0&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&exporter.listener=injvm&file-cache=true&generic=false&getAndListenInstanceMetadata.1.callback=true&getAndListenInstanceMetadata.return=true&getAndListenInstanceMetadata.sent=true&group=crm-dubbo&interface=org.apache.dubbo.metadata.MetadataService&methods=exportInstanceMetadata,getAndListenInstanceMetadata,getExportedServiceURLs,getExportedURLs,getInstanceMetadataChangedListenerMap,getMetadataInfo,getMetadataInfos,getMetadataURL,getServiceDefinition,getSubscribedURLs,isMetadataService,serviceName,toSortedStrings,version&pid=1&prefer.serialization=hessian2,fastjson2®ister=false&
register-mode=all&release=3.2.6&retries=0&revision=3.2.6&serialization=hessian2&serialize.check.status=WARN&service.filter=-default&shutwait=30000&side=provider×tamp=1695125322905&version=1.0.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.058 INFO [SpringApplicationShutdownHook] org.apache.dubbo.qos.server.Server Line:133 - [DUBBO] qos-server stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.062 INFO [SpringApplicationShutdownHook] org.apache.dubbo.qos.server.Server Line:133 - [DUBBO] qos-server stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.064 INFO [SpringApplicationShutdownHook] org.apache.dubbo.qos.server.Server Line:133 - [DUBBO] qos-server stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.065 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:181 - [DUBBO] Destroying protocol [TripleProtocol] ..., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.066 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.IDemonstrationCenterService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.cmc.console.api.service.IDemonstrationCenterService&methods=addIndustry,delIndustry,selectAll,updateIndustry&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125285116&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.067 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cc.dispatcher.api.service.ICallProcessService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&group=public_crm&interface=com.example.cc.dispatcher.api.service.ICallProcessService&methods=processPhoneParamBeforeCall,processTtsParamBeforeCall&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=4.7.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider&timeout=30000×tamp=1695125298321&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.068 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.ICustomerService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.cmc.console.api.service.ICustomerService&methods=getCustomerBelongsToAgentGroup&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125296519&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.068 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.ILogService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.cmc.console.api.service.ILogService&methods=saveLog&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125286490&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.069 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.INotifyMessageService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.cmc.console.api.service.INotifyMessageService&methods=insertNotifyMessage&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125287425&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.070 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.ITaskImportBatchService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.cmc.console.api.service.ITaskImportBatchService&methods=listCallTaskImportBatch,saveTaskImportBatch&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125297434&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.070 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.ICallRecordService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&getAnswerNumberCallItemCount.return=true&getAnswerNumberCallItemCount.sent=true&getAnswerNumberCallItemCount.timeout=60000&getCallTaskItemCount.return=true&getCallTaskItemCount.sent=true&getCallTaskItemCount.timeout=60000&interface=com.example.cmc.console.api.service.ICallRecordService&methods=getAgentCallStatistics,getAnswerNumberCallItemCount,getAudio,getAudioURL,getCallDialogRecords,getCallStatistics,getCallTaskItemCount,getEntCallTaskByTemplateCode,queryCallRecordList&pid=1&prefer.serialization=hessian2,fastjson2®ister-mo
de=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider&timeout=10000×tamp=1695125299237&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.071 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/org.apache.dubbo.metadata.MetadataService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=0&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&getAndListenInstanceMetadata.1.callback=true&getAndListenInstanceMetadata.return=true&getAndListenInstanceMetadata.sent=true&group=crm-dubbo&interface=org.apache.dubbo.metadata.MetadataService&methods=exportInstanceMetadata,getAndListenInstanceMetadata,getExportedServiceURLs,getExportedURLs,getInstanceMetadataChangedListenerMap,getMetadataInfo,getMetadataInfos,getMetadataURL,getServiceDefinition,getSubscribedURLs,isMetadataService,serviceName,toSortedStrings,version&pid=1&prefer.serialization=hessian2,fastjson2®ister=false®ister-mode=al
l&release=3.2.6&retries=0&revision=3.2.6&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&service.filter=-default&shutwait=30000&side=provider×tamp=1695125322905&version=1.0.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.072 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.ICallTaskService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.cmc.console.api.service.ICallTaskService&methods=addCalledPhonesToTask,getCallTaskById,getCallTaskVarDTOById,getExportVarNames,selectCallRecordByCallId,selectCountCallTaskByDepartmentId,tryCall&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125291799&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.072 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:110 - [DUBBO] Destroy reference: tri://10.244.23.158:50051/com.example.cmc.console.api.service.IOperateResultService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&check.serializable=true&delay=-1&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&interface=com.example.cmc.console.api.service.IOperateResultService&methods=saveOperateResult,selectOperateResultById,updateOperateResult&pid=1&prefer.serialization=hessian2,fastjson2®ister-mode=all&release=3.2.6&retries=0&revision=5.15.4-SNAPSHOT&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&shutwait=30000&side=provider×tamp=1695125300156&version=1.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.073 INFO [SpringApplicationShutdownHook] org.apache.dubbo.qos.server.Server Line:133 - [DUBBO] qos-server stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.074 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.protocol.tri.TripleProtocol Line:181 - [DUBBO] Destroying protocol [TripleProtocol] ..., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.074 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultModuleDeployer Line:368 - [DUBBO] Dubbo Module[1.1.0] is stopping., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.081 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultModuleDeployer Line:378 - [DUBBO] Dubbo Module[1.1.0] has stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.086 INFO [SpringApplicationShutdownHook] org.apache.dubbo.registry.support.RegistryManager Line:101 - [DUBBO] Close all registries [nacos://nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?REGISTRY_CLUSTER=internal-metadata-registry&application=crm-dubbo&check=true&check.serializable=true&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&password=nacos&pid=1®ister=true®ister-mode=all&release=3.2.6&serialize.check.status=WARN&shutwait=30000&subscribe=true&timeout=5000&username=nacos, nacos://nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?application=crm-dubbo&check=true&check.serializable=true&du
bbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&password=nacos&pid=1®ister=false®ister-mode=all&release=3.2.6&serialize.check.status=WARN&shutwait=30000&subscribe=true&timeout=5000&username=nacos, nacos://nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?application=crm-dubbo&check=true&check.serializable=true&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&password=nacos&pid=1®ister=true®ister-mode=all&release=3.2.6&serialize.check.status=WARN&shutwait=30000&subscribe=true&timeout=5000&username=nacos, nacos://nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848/org.apache.dubbo.registry.Registr
yService?application=crm-dubbo&check=true&check.serializable=true&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&password=nacos&pid=1®ister=false®ister-mode=all&release=3.2.6&serialize.check.status=WARN&shutwait=30000&subscribe=true&timeout=5000&username=nacos], dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.142 INFO [SpringApplicationShutdownHook] org.apache.dubbo.registry.nacos.NacosRegistry Line:576 - [DUBBO] Destroy registry:nacos://nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848/org.apache.dubbo.registry.RegistryService?application=crm-dubbo&check=true&check.serializable=true&dubbo=2.0.2&executor-management-mode=isolation&file-cache=true&interface=org.apache.dubbo.registry.RegistryService&password=nacos&pid=1®ister=false®ister-mode=all&release=3.2.6&serialize.check.status=WARN&shutwait=30000&subscribe=true&timeout=5000&username=nacos, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.144 INFO [SpringApplicationShutdownHook] org.apache.dubbo.registry.nacos.NacosRegistry Line:430 - [DUBBO] Unsubscribe: provider://10.244.23.158:50051/org.apache.dubbo.metadata.MetadataService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&category=configurators&check=false&check.serializable=true&delay=0&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&getAndListenInstanceMetadata.1.callback=true&getAndListenInstanceMetadata.return=true&getAndListenInstanceMetadata.sent=true&group=crm-dubbo&interface=org.apache.dubbo.metadata.MetadataService&methods=exportInstanceMetadata,getAndListenInstanceMetadata,getExportedServiceURLs,getExportedURLs,getInstanceMetadataChangedListenerMap,getMetadataInfo,getMetadataInfos,getMetadataURL,getServiceDefinition,getSubscribedURLs,isMetadataService,serviceName,toSortedStrings,version&pid=1&prefer.serialization=hessian2,fastjson2&
register=false®ister-mode=all&release=3.2.6&retries=0&revision=3.2.6&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&service.filter=-default&shutwait=30000&side=provider×tamp=1695125322905&version=1.0.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.181 INFO [SpringApplicationShutdownHook] org.apache.dubbo.registry.nacos.NacosRegistry Line:603 - [DUBBO] Destroy unsubscribe url provider://10.244.23.158:50051/org.apache.dubbo.metadata.MetadataService?anyhost=true&application=crm-dubbo&background=false&bind.ip=10.244.23.158&bind.port=50051&category=configurators&check=false&check.serializable=true&delay=0&deprecated=false&dubbo=2.0.2&dynamic=true&executor-management-mode=isolation&file-cache=true&generic=false&getAndListenInstanceMetadata.1.callback=true&getAndListenInstanceMetadata.return=true&getAndListenInstanceMetadata.sent=true&group=crm-dubbo&interface=org.apache.dubbo.metadata.MetadataService&methods=exportInstanceMetadata,getAndListenInstanceMetadata,getExportedServiceURLs,getExportedURLs,getInstanceMetadataChangedListenerMap,getMetadataInfo,getMetadataInfos,getMetadataURL,getServiceDefinition,getSubscribedURLs,isMetadataService,serviceName,toSortedStrings,version&pid=1&prefer.serialization=hessian2
,fastjson2®ister=false®ister-mode=all&release=3.2.6&retries=0&revision=3.2.6&serialization=hessian2&serialize.check.status=WARN&service-name-mapping=true&service.filter=-default&shutwait=30000&side=provider×tamp=1695125322905&version=1.0.0, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.200 INFO [SpringApplicationShutdownHook] org.apache.dubbo.common.threadpool.manager.DefaultExecutorRepository Line:403 - [DUBBO] destroying application executor repository .., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.212 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultApplicationDeployer Line:1256 - [DUBBO] Dubbo Application[1.1](crm-dubbo) has stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.218 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.model.FrameworkModel Line:115 - [DUBBO] Destroying default framework model: Dubbo Framework[1], dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.219 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.model.FrameworkModel Line:120 - [DUBBO] Dubbo Framework[1] is destroying ..., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.220 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultApplicationDeployer Line:1242 - [DUBBO] Dubbo Application[1.0](DUBBO_INTERNAL_APPLICATION) is stopping., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.221 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultModuleDeployer Line:368 - [DUBBO] Dubbo Module[1.0.0] is stopping., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.221 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultModuleDeployer Line:378 - [DUBBO] Dubbo Module[1.0.0] has stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.222 INFO [SpringApplicationShutdownHook] org.apache.dubbo.registry.support.RegistryManager Line:101 - [DUBBO] Close all registries [], dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.223 INFO [SpringApplicationShutdownHook] org.apache.dubbo.common.threadpool.manager.DefaultExecutorRepository Line:403 - [DUBBO] destroying application executor repository .., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.224 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.deploy.DefaultApplicationDeployer Line:1256 - [DUBBO] Dubbo Application[1.0](DUBBO_INTERNAL_APPLICATION) has stopped., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.226 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.model.FrameworkModel Line:135 - [DUBBO] Dubbo Framework[1] is destroyed, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.227 INFO [SpringApplicationShutdownHook] org.apache.dubbo.rpc.model.FrameworkModel Line:327 - [DUBBO] Reset global default framework from Dubbo Framework[1] to null, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.227 INFO [SpringApplicationShutdownHook] org.apache.dubbo.common.resource.GlobalResourcesRepository Line:107 - [DUBBO] Destroying global resources ..., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.257 INFO [SpringApplicationShutdownHook] org.apache.dubbo.common.resource.GlobalResourcesRepository Line:145 - [DUBBO] Dubbo is completely destroyed, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.258 INFO [SpringApplicationShutdownHook] org.apache.dubbo.common.threadpool.manager.FrameworkExecutorRepository Line:189 - [DUBBO] destroying framework executor repository .., dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.278 INFO [SpringApplicationShutdownHook] org.apache.dubbo.config.spring.context.DubboSpringInitializer Line:80 - [DUBBO] Unbind Dubbo Module[1.1.1] from spring container: org.springframework.beans.factory.support.DefaultListableBeanFactory@2287395, dubbo version: 3.2.6, current host: 10.244.23.158
2023-09-19 20:34:30.569 INFO [SpringApplicationShutdownHook] org.springframework.boot.web.embedded.undertow.UndertowWebServer Line:295 - Commencing graceful shutdown. Waiting for active requests to complete
2023-09-19 20:34:30.576 INFO [SpringApplicationShutdownHook] org.springframework.boot.web.embedded.undertow.UndertowWebServer Line:305 - Graceful shutdown complete
2023-09-19 20:34:30.662 INFO [SpringApplicationShutdownHook] org.springframework.scheduling.quartz.SchedulerFactoryBean Line:847 - Shutting down Quartz Scheduler
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] chaoyoung commented on issue #13081: Directory of type RegistryDirectory already destroyed for service xxx while dubbo graceful shutdown
Posted by "chaoyoung (via GitHub)" <gi...@apache.org>.
chaoyoung commented on issue #13081:
URL: https://github.com/apache/dubbo/issues/13081#issuecomment-1728680272
目前通过外部先切掉入口HTTP流量解决了此问题。
Spring Boot `application.yml` 配置graceful shutdown
```yml
server:
shutdown: graceful
spring:
lifecycle:
timeout-per-shutdown-phase: 30s
```
应用程序提供实例下线的接口
```java
@RestController
@RequiredArgsConstructor
@Slf4j
public class GracefulShutdownController {
private final NacosAutoServiceRegistration nacosAutoServiceRegistration;
@PostMapping("/instance/offline")
public void instanceOffline() {
nacosAutoServiceRegistration.destroy();
log.info("instance offline from nacos.");
}
}
```
k8s preStop 通过curl调用实例下线的接口，将入口http流量先断掉，然后等Spring和Dubbo graceful shutdown，最多等待30s。
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: xxx-backend
namespace: default
labels:
app: xxx-backend
spec:
replicas: 1
selector:
matchLabels:
app: xxx-backend
template:
metadata:
labels:
app: xxx-backend
spec:
containers:
- name: xxx-backend
image: registry.cn-hangzhou.aliyuncs.com/namespace/xxx-backend:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
name: http
protocol: TCP
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "curl -X POST localhost:8080/instance/offline"]
terminationGracePeriodSeconds: 30
```
因为使用了Spring Cloud Gateway作为后端微服务网关，所以Gateway这边也需要在实例下线的时候清理缓存的操作。
```java
/**
* 订阅Nacos通知
* 接收Nacos推送的微服务上下线实例信息
*/
@Component
@Slf4j
public class NacosInstancesChangeEventListener extends Subscriber<InstancesChangeEvent> {
@Resource
private CacheManager caffeineLoadBalancerCacheManager;
@PostConstruct
public void registerToNotifyCenter() {
NotifyCenter.registerSubscriber(this);
}
@Override
public void onEvent(InstancesChangeEvent event) {
log.info("Spring Gateway 接收实例刷新事件：{}, 开始刷新缓存", JacksonUtils.toJson(event));
Cache cache = caffeineLoadBalancerCacheManager.getCache(SERVICE_INSTANCE_CACHE_NAME);
if (cache != null) {
cache.evict(event.getServiceName());
}
log.info("Spring Gateway 实例刷新完成");
}
@Override
public Class<? extends com.alibaba.nacos.common.notify.Event> subscribeType() {
return InstancesChangeEvent.class;
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] chaoyoung commented on issue #13081: Directory of type RegistryDirectory already destroyed for service xxx while dubbo graceful shutdown
Posted by "chaoyoung (via GitHub)" <gi...@apache.org>.
chaoyoung commented on issue #13081:
URL: https://github.com/apache/dubbo/issues/13081#issuecomment-1729010738
实测通过preStop hook直接调用Nacos API`curl -X PUT "http://nacos-headless.nacos.svc.cluster.local:8848/nacos/v1/ns/instance?serviceName=xxx-backend&namespaceId=&clusterName=DEFAULT&groupName=DEFAULT_GROUP&ip=$POD_IP&port=8080&ephemeral=true&enabled=false&username=$NACOS_USERNAME&password=$NACOS_PASSWORD"`完成服务下线，避免每个微服务还需要单独增加服务下线接口。
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: xxx-backend
namespace: default
labels:
app: xxx-backend
spec:
replicas: 1
selector:
matchLabels:
app: xxx-backend
template:
metadata:
labels:
app: xxx-backend
spec:
containers:
- name: xxx-backend
image: registry.cn-hangzhou.aliyuncs.com/namespace/xxx-backend:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
name: http
protocol: TCP
env:
- name: NACOS_SERVER_ADDR
valueFrom:
configMapKeyRef:
name: common-cm
key: nacos.server-addr
- name: NACOS_USERNAME
valueFrom:
configMapKeyRef:
name: common-cm
key: nacos.username
- name: NACOS_PASSWORD
valueFrom:
configMapKeyRef:
name: common-cm
key: nacos.password
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
lifecycle:
preStop:
exec:
command:
- bash
- -c
- curl -X PUT "http://nacos-headless.nacos.svc.cluster.local:8848/nacos/v1/ns/instance?serviceName=xxx-backend&namespaceId=&clusterName=DEFAULT&groupName=DEFAULT_GROUP&ip=$POD_IP&port=8080&ephemeral=true&enabled=false&username=$NACOS_USERNAME&password=$NACOS_PASSWORD"
terminationGracePeriodSeconds: 30
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] Directory of type RegistryDirectory already destroyed for service xxx while dubbo graceful shutdown [dubbo]
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ closed issue #13081: Directory of type RegistryDirectory already destroyed for service xxx while dubbo graceful shutdown
URL: https://github.com/apache/dubbo/issues/13081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #13081: Directory of type RegistryDirectory already destroyed for service xxx while dubbo graceful shutdown
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #13081:
URL: https://github.com/apache/dubbo/issues/13081#issuecomment-1735409187
1. 应用下线的时候，需要把入口流量现摘除（包括 Http、Dubbo 等）
2. Dubbo 的下线可以通过调用 http://127.0.0.1:22222/offine 实现
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] Directory of type RegistryDirectory already destroyed for service xxx while dubbo graceful shutdown [dubbo]
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #13081:
URL: https://github.com/apache/dubbo/issues/13081#issuecomment-1987583130
No news is good news. Please feel free to create a new issue if you have any question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] Directory of type RegistryDirectory already destroyed for service xxx while dubbo graceful shutdown [dubbo]
Posted by "yipixiaofeiyang (via GitHub)" <gi...@apache.org>.
yipixiaofeiyang commented on issue #13081:
URL: https://github.com/apache/dubbo/issues/13081#issuecomment-2128386003
V3.0.8的时候, 还是没问题的, 升到3.2.11后, 发现就开始报
Directory of type ServiceDiscoveryRegistryDirectory already destroyed for service 'xxxx'了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org