You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2022/11/21 02:32:59 UTC
[GitHub] [dubbo] HTFX-Cloud opened a new issue, #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
HTFX-Cloud opened a new issue, #10965:
URL: https://github.com/apache/dubbo/issues/10965
<!-- If you need to report a security issue please visit https://github.com/apache/dubbo/security/policy -->
- [ ] I have searched the [issues](https://github.com/apache/dubbo/issues) of this repository and believe that this is not a duplicate.
### Environment
* Dubbo version: 2.7.8
* Operating System version: CentOS 7.6.1810
* Java version: 1.8.0_181
* Nacos version: 1.3.2
### Steps to reproduce this issue
1. 有四个节点，分别为A\B\C\D。每个节点既是服务消费者也是提供者。
2. 在某一时刻，A节点调用XX服务时，出现B节点的isAvailable状态被置为false。但从B节点看invoker状态负载正常
3. 出现异常状态时，A节点的日志存在【Consumer is shutting down】，关联的https://github.com/apache/dubbo/issues/6294
4. 目前A节点负载B异常，但B负载A正常的情况。目前我们采用A节点直连B节点，重新在A上创建对B的netty连接。
5. 有个疑问就是，为什么会在A节点上负载不到某些节点，但是某些节点却能负载到A节点？在A上负载的invoker中的isAvailable状态什么时候可能被置为false？是否与dubbo的时间轮有关https://zhuanlan.zhihu.com/p/414441500？是否与nacos1.3.2版本的线程模型存在大量nacos客户端连接导致？希望有人能看见，帮忙一起分析。谢谢！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1321421374
从日志上看是 provider 到 nacos 的心跳断开了，导致 nacos 推送该 provider 下线，但是有在途请求然后报错
可以检查下为什么 nacos 注册中心会推送下线信息
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] HTFX-Cloud commented on issue #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
Posted by GitBox <gi...@apache.org>.
HTFX-Cloud commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1321403386
从10点开始出现[Dubbo-thread-2] ~ WARN org.apache.dubbo.remoting.exchange.support.DefaultFuture 177 received - [DUBBO] The timeout response finally returned at 2022-10-19 10:01:09.345, response status is 20, channel: /10.177.56.133:43182 -> /10.177.56.133:20882, please check provider side for detailed result., dubbo version: 2.7.8, current host: 161.2.56.133
以及error的日志（该日志倒排取的，需要注意）
10-19 10:27:32,600 [pool-7-thread-1700] ~c885fe8c32434010a0eb9f06892c1b6f INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 675 doReleaseRes - NODE[c885fe8c32434010a0eb9f06892c1b6f] 执行任务释放资源步骤：释放资源！
10-19 10:27:32,617 [pool-7-thread-2957] ~4031a4934f6c47cab380c66e2e46f4df INFO a.b.c.bd.logic.impl.TodoTaskOperationService 269 processTodoTask - 处理代办时，workflowKey=03a7a64a0ce14b9b95d264036d964d89,workflowCurrentKey=69A7FA817268477B9C0DBF3686396C4F,nodeKey=fee5cfb39e9748de9fa152f55bc12b18,taskRunId=d1555c00314c4a759d333316327cecac对应的任务启动时的项目信息为{projectId=null, spaceId=0, WfbloodRerunInfo=}
10-19 10:27:32,627 [pool-7-thread-4967] ~357eeb0785e2443a8d0f3594c8307ad5 DEBUG a.b.c.bd.dao.impl.TaskOperationDao 1503 getAllStatus - getAllStatus:[ ]
10-19 10:27:32,627 [pool-7-thread-1405] ~720d2d06a8744456b4d740d5cd1be846 INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 410 doBegin - NODE[720d2d06a8744456b4d740d5cd1be846] 执行任务开始步骤：初始化任务变量！
... 3 more
at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:132) ~[nacos-client-1.3.0.jar:?]
at org.apache.dubbo.registry.nacos.NacosRegistry.lambda$subscribeEventListener$10(NacosRegistry.java:507) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.nacos.NacosRegistry.notifySubscriber(NacosRegistry.java:529) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.support.FailbackRegistry.notify(FailbackRegistry.java:398) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.support.FailbackRegistry.doNotify(FailbackRegistry.java:407) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.support.AbstractRegistry.notify(AbstractRegistry.java:426) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.notify(RegistryDirectory.java:254) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.refreshOverrideAndInvoker(RegistryDirectory.java:271) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.refreshInvoker(RegistryDirectory.java:336) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.destroyUnusedInvokers(RegistryDirectory.java:580) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.InvokerWrapper.destroy(InvokerWrapper.java:61) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.destroy(ProtocolFilterWrapper.java:131) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.destroy(ListenerInvokerWrapper.java:89) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.destroy(AsyncToSyncInvoker.java:96) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.destroy(DubboInvoker.java:153) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.close(ReferenceCountExchangeClient.java:164) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.close(LazyConnectExchangeClient.java:204) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.close(HeaderExchangeClient.java:144) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:177) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:152) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.closeChannel(DefaultFuture.java:147) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.threadpool.ThreadlessExecutor.shutdownNow(ThreadlessExecutor.java:164) ~[dubbo-2.7.8.jar:2.7.8]
Caused by: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.
... 57 more
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:51) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:89) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:61) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:181) ~[dubbo-2.7.8.jar:2.7.8]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_181]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.run(DispatcherNodeInstanceHandler.java:192) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:376) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBegin(DispatcherNodeInstanceHandler.java:446) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:380) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doCompeteRes(DispatcherNodeInstanceHandler.java:478) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:384) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBusiness(DispatcherNodeInstanceHandler.java:596) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:388) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReceiveVar(DispatcherNodeInstanceHandler.java:666) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:392) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReleaseRes(DispatcherNodeInstanceHandler.java:681) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:396) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doTrigger(DispatcherNodeInstanceHandler.java:749) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:400) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doEnd(DispatcherNodeInstanceHandler.java:794) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService$$EnhancerBySpringCGLIB$$1e7ef4d6.processBatchTodoTasksAsyncWithRerunTodos(<generated>) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at a.b.c.bd.logic.impl.TodoTaskOperationService$$FastClassBySpringCGLIB$$29b976c.invoke(<generated>) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasksAsyncWithRerunTodos(TodoTaskOperationService.java:961) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasks(TodoTaskOperationService.java:1065) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService.processTodoTask(TodoTaskOperationService.java:415) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:78) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:135) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at org.apache.dubbo.common.bytecode.proxy0$$EnhancerBySpringCGLIB$$75f0c32c.executeTask(<generated>) ~[dubbo-2.7.8.jar:2.7.8]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.apache.dubbo.common.bytecode.proxy0$$FastClassBySpringCGLIB$$c197671e.invoke(<generated>) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.bytecode.proxy0.executeTask(proxy0.java) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:88) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:260) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:113) ~[dubbo-2.7.8.jar:2.7.8]
10-19 10:27:43,391 [pool-7-thread-4442] ~416ce361b8e9466f894a46a6e34adace ERROR a.b.c.bd.manager.TaskExecutionService 199 executeTask - 1次调用后非文件存储异常信息 org.apache.dubbo.rpc.RpcException: Failed to invoke the method executeTask in the service a.b.c.bd.api.IEngineService. Tried 1 times of the providers [10.177.56.133:20882] (1/4) from the registry 10.177.56.130:9908 on the consumer 161.2.56.133 using the dubbo version 2.7.8. Last error is: Fail to invoke remote method: executeTask, provider: dubbo://10.177.56.133:20882/engineService?anyhost=true&application=dbd-app&category=providers&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=public&init=true&interface=a.b.c.bd.api.IEngineService&metadata-type=remote&methods=reRunTask,executeTask&path=engineService&payload=104857600&pid=2378&protocol=dubbo&qos.enable=false®ister.ip=161.2.56.133&release=2.7.8&remote.application=dbd-app&retries=0&revision=5.3-MASTER-20220608.095739-141&side=co
nsumer&sticky=false&timeout=30000×tamp=1655987866520&version=5.0, cause: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.
10-19 10:27:32,632 [pool-7-thread-305] ~db4ed7dab49b42dcaa809c8f212a2e5e DEBUG a.b.c.bd.consumer.PrepareForTaskExecutor 110 prepareExecutionParameter - DataOps: key=projectId,value=null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
Re: [I] nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡 [dubbo]
Posted by "CrazyHZM (via GitHub)" <gi...@apache.org>.
CrazyHZM closed issue #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
URL: https://github.com/apache/dubbo/issues/10965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] HTFX-Cloud commented on issue #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
Posted by GitBox <gi...@apache.org>.
HTFX-Cloud commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1321409085
> > 出现异常状态时，A节点的日志存在【Consumer is shutting down】，关联的#6294
>
> 这个的完整报错堆栈可以贴一下吗
从10点开始出现Warn级别日志
[Dubbo-thread-2] ~ WARN org.apache.dubbo.remoting.exchange.support.DefaultFuture 177 received - [DUBBO] The timeout response finally returned at 2022-10-19 10:01:09.345, response status is 20, channel: /10.177.56.133:43182 -> /10.177.56.133:20882, please check provider side for detailed result., dubbo version: 2.7.8, current host: 161.2.56.133
以及error的完整信息
10-19 10:27:32,632 [pool-7-thread-305] ~db4ed7dab49b42dcaa809c8f212a2e5e DEBUG a.b.c.bd.consumer.PrepareForTaskExecutor 110 prepareExecutionParameter - DataOps: key=projectId,value=null
10-19 10:27:43,391 [pool-7-thread-4442] ~416ce361b8e9466f894a46a6e34adace ERROR a.b.c.bd.manager.TaskExecutionService 199 executeTask - 1次调用后非文件存储异常信息 org.apache.dubbo.rpc.RpcException: Failed to invoke the method executeTask in the service a.b.c.bd.api.IEngineService. Tried 1 times of the providers [10.177.56.133:20882] (1/4) from the registry 10.177.56.130:9908 on the consumer 161.2.56.133 using the dubbo version 2.7.8. Last error is: Fail to invoke remote method: executeTask, provider: dubbo://10.177.56.133:20882/engineService?anyhost=true&application=dbd-app&category=providers&check=false&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&group=public&init=true&interface=a.b.c.bd.api.IEngineService&metadata-type=remote&methods=reRunTask,executeTask&path=engineService&payload=104857600&pid=2378&protocol=dubbo&qos.enable=false®ister.ip=161.2.56.133&release=2.7.8&remote.application=dbd-app&retries=0&revision=5.3-MASTER-20220608.095739-141&side=co
nsumer&sticky=false&timeout=30000×tamp=1655987866520&version=5.0, cause: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:113) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:260) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.interceptor.ClusterInterceptor.intercept(ClusterInterceptor.java:47) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.wrapper.AbstractCluster$InterceptorInvokerNode.invoke(AbstractCluster.java:92) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:88) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:74) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.bytecode.proxy0.executeTask(proxy0.java) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.common.bytecode.proxy0$$FastClassBySpringCGLIB$$c197671e.invoke(<generated>) ~[dubbo-2.7.8.jar:2.7.8]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.apache.dubbo.common.bytecode.proxy0$$EnhancerBySpringCGLIB$$75f0c32c.executeTask(<generated>) ~[dubbo-2.7.8.jar:2.7.8]
at a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:135) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.manager.TaskExecutionService.executeTask(TaskExecutionService.java:78) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService.processTodoTask(TodoTaskOperationService.java:415) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasks(TodoTaskOperationService.java:1065) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService.processBatchTodoTasksAsyncWithRerunTodos(TodoTaskOperationService.java:961) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.logic.impl.TodoTaskOperationService$$FastClassBySpringCGLIB$$29b976c.invoke(<generated>) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98) ~[spring-tx-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:93) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688) ~[spring-aop-5.1.8.RELEASE.jar:5.1.8.RELEASE]
at a.b.c.bd.logic.impl.TodoTaskOperationService$$EnhancerBySpringCGLIB$$1e7ef4d6.processBatchTodoTasksAsyncWithRerunTodos(<generated>) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doEnd(DispatcherNodeInstanceHandler.java:794) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:400) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doTrigger(DispatcherNodeInstanceHandler.java:749) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:396) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReleaseRes(DispatcherNodeInstanceHandler.java:681) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:392) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doReceiveVar(DispatcherNodeInstanceHandler.java:666) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:388) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBusiness(DispatcherNodeInstanceHandler.java:596) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:384) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doCompeteRes(DispatcherNodeInstanceHandler.java:478) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:380) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doNextStep(DispatcherNodeInstanceHandler.java:1621) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.doBegin(DispatcherNodeInstanceHandler.java:446) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.exec(DispatcherNodeInstanceHandler.java:376) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at a.b.c.bd.handler.DispatcherNodeInstanceHandler.run(DispatcherNodeInstanceHandler.java:192) ~[dbd-service-5.3-MASTER-20220608.095809-138.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_181]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[?:1.8.0_181]
at org.apache.dubbo.rpc.AsyncRpcResult.get(AsyncRpcResult.java:181) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.invoke(AsyncToSyncInvoker.java:61) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:78) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:89) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:51) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:69) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:83) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:56) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:82) ~[dubbo-2.7.8.jar:2.7.8]
... 57 more
Caused by: java.lang.IllegalStateException: Consumer is shutting down and this call is going to be stopped without receiving any result, usually this is called by a slow provider instance or bad service implementation.
at org.apache.dubbo.common.threadpool.ThreadlessExecutor.shutdownNow(ThreadlessExecutor.java:164) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.DefaultFuture.closeChannel(DefaultFuture.java:147) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:152) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeChannel.close(HeaderExchangeChannel.java:177) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.remoting.exchange.support.header.HeaderExchangeClient.close(HeaderExchangeClient.java:144) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.LazyConnectExchangeClient.close(LazyConnectExchangeClient.java:204) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.ReferenceCountExchangeClient.close(ReferenceCountExchangeClient.java:164) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.dubbo.DubboInvoker.destroy(DubboInvoker.java:153) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.AsyncToSyncInvoker.destroy(AsyncToSyncInvoker.java:96) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.listener.ListenerInvokerWrapper.destroy(ListenerInvokerWrapper.java:89) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.ProtocolFilterWrapper$1.destroy(ProtocolFilterWrapper.java:131) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.rpc.protocol.InvokerWrapper.destroy(InvokerWrapper.java:61) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.destroyUnusedInvokers(RegistryDirectory.java:580) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.refreshInvoker(RegistryDirectory.java:336) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.refreshOverrideAndInvoker(RegistryDirectory.java:271) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.integration.RegistryDirectory.notify(RegistryDirectory.java:254) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.support.AbstractRegistry.notify(AbstractRegistry.java:426) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.support.FailbackRegistry.doNotify(FailbackRegistry.java:407) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.support.FailbackRegistry.notify(FailbackRegistry.java:398) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.nacos.NacosRegistry.notifySubscriber(NacosRegistry.java:529) ~[dubbo-2.7.8.jar:2.7.8]
at org.apache.dubbo.registry.nacos.NacosRegistry.lambda$subscribeEventListener$10(NacosRegistry.java:507) ~[dubbo-2.7.8.jar:2.7.8]
at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:132) ~[nacos-client-1.3.0.jar:?]
... 3 more
10-19 10:27:32,627 [pool-7-thread-1405] ~720d2d06a8744456b4d740d5cd1be846 INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 410 doBegin - NODE[720d2d06a8744456b4d740d5cd1be846] 执行任务开始步骤：初始化任务变量！
10-19 10:27:32,627 [pool-7-thread-4967] ~357eeb0785e2443a8d0f3594c8307ad5 DEBUG a.b.c.bd.dao.impl.TaskOperationDao 1503 getAllStatus - getAllStatus:[ ]
10-19 10:27:32,617 [pool-7-thread-2957] ~4031a4934f6c47cab380c66e2e46f4df INFO a.b.c.bd.logic.impl.TodoTaskOperationService 269 processTodoTask - 处理代办时，workflowKey=03a7a64a0ce14b9b95d264036d964d89,workflowCurrentKey=69A7FA817268477B9C0DBF3686396C4F,nodeKey=fee5cfb39e9748de9fa152f55bc12b18,taskRunId=d1555c00314c4a759d333316327cecac对应的任务启动时的项目信息为{projectId=null, spaceId=0, WfbloodRerunInfo=}
10-19 10:27:32,600 [pool-7-thread-1700] ~c885fe8c32434010a0eb9f06892c1b6f INFO a.b.c.bd.handler.DispatcherNodeInstanceHandler 675 doReleaseRes - NODE[c885fe8c32434010a0eb9f06892c1b6f] 执行任务释放资源步骤：释放资源！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] HTFX-Cloud commented on issue #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
Posted by GitBox <gi...@apache.org>.
HTFX-Cloud commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1321559332
> 从日志上看是 provider 到 nacos 的心跳断开了，导致 nacos 推送该 provider 下线，但是有在途请求然后报错
>
> 可以检查下为什么 nacos 注册中心会推送下线信息
我看了一下nacos集群（130、131、132）三个节点上10.19日的naming.log文件。其中在10:27:04.914和10:27:20.508两个时间三个节点都存在
2022-10-19 10:27:04.914 INFO [com.alibaba.nacos.client.naming.updater:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IEngineService:5.0:public -> [{"clusterName":"DEFAULT","enabled":true,"ephemeral":true,"healthy":true,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000,"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IEngineService:5.0:public","ip":"10.177.56.133","ipDeleteTimeout":30000,"metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"com.bonc.dataplatform.buw.dispatcher.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"p
roviders","anyhost":"true","group":"public","timestamp":"1655987866520"},"port":20882,"serviceName":"DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IEngineService:5.0:public","weight":1.0}]
和
2022-10-19 10:27:20.508 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService:5.0:public -> [{"clusterName":"DEFAULT","enabled":true,"ephemeral":true,"healthy":true,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000,"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService:5.0:public","ip":"10.177.56.133","ipDeleteTimeout":30000,"metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,startWorkflow,startScheduleWorkflow,executeTaskForTemp,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflo
ws,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987866761"},"port":20882,"serviceName":"DEFAULT_GROUP@@providers:com.bonc.dataplatform.buw.dispatcher.api.IWorkflowWebService:5.0:public","weight":1.0}]摘除133节点（**也就是上文提到负载调用不平衡的A节点**）
并且在nacos集群中的132节点存在如下的日志
2022-10-19 10:27:05.066 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] received push data: {"type":"dom","data":"{\"hosts\":[{\"ip\":\"10.177.56.132\",\"port\":20882,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"53008\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987760581\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":
\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.134\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"31262\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987989629\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceNam
e\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.135\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"56132\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655988109440\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@pro
viders:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true}],\"dom\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"name\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"cacheMillis\":10000,\"lastRefTime\":1666146425065,\"checksum\":\"9190a7c1dcdb5fce2375f5282bd116ae\",\"useSpecifiedURL\":false,\"clusters\":\"\",\"env\":\"\",\"metadata\":{}}","lastRefTime":56135268660146131} from /10.177.56.132
2022-10-19 10:27:05.066 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] received push data: {"type":"dom","data":"{\"hosts\":[{\"ip\":\"10.177.56.132\",\"port\":20882,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"53008\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987760581\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":
\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.134\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"31262\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987989629\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceNam
e\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.135\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"reRunTask,executeTask\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"56132\",\"interface\":\"a.b.c.bd.api.IEngineService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"engineService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655988109440\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@pro
viders:a.b.c.bd.api.IEngineService:5.0:public\",\"ephemeral\":true}],\"dom\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"name\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public\",\"cacheMillis\":10000,\"lastRefTime\":1666146425067,\"checksum\":\"9190a7c1dcdb5fce2375f5282bd116ae\",\"useSpecifiedURL\":false,\"clusters\":\"\",\"env\":\"\",\"metadata\":{}}","lastRefTime":56135268660146131} from /10.177.56.132
2022-10-19 10:27:05.085 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.133","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"165598786652
0"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:05.085 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.133","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"165598786652
0"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:05.086 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] current ips:(3) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.132","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"53008","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987760
581"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.134","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"31262","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987989629"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatT
imeOut":15000},{"instanceId":"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.135","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"56132","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655988109440"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:05.529 WARN [com.alibaba.nacos.naming.push.receiver:client.naming] write DEFAULT_GROUP%40%40providers%3Aa.b.c.bd.api.IEngineService%3A5.0%3Apublic conflict;retry time: 1
2022-10-19 10:27:05.529 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] current ips:(3) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public -> [{"instanceId":"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.132","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"53008","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987760
581"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.134","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"31262","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987989629"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatT
imeOut":15000},{"instanceId":"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","ip":"10.177.56.135","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IEngineService:5.0:public","metadata":{"side":"provider","methods":"reRunTask,executeTask","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"56132","interface":"a.b.c.bd.api.IEngineService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"engineService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655988109440"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:20.858 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] received push data: {"type":"dom","data":"{\"hosts\":[{\"ip\":\"10.177.56.132\",\"port\":20882,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"53008\",\"interface\":\"a.b.
c.bd.api.IWorkflowWebService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"WorkflowWebService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987760900\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.134\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,d
ealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"31262\",\"interface\":\"a.b.c.bd.api.IWorkflowWebService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revision\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"WorkflowWebService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655987989873\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebSe
rvice:5.0:public\",\"ephemeral\":true},{\"ip\":\"10.177.56.135\",\"port\":20881,\"valid\":true,\"healthy\":true,\"marked\":false,\"instanceId\":\"10.177.56.135#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"metadata\":{\"side\":\"provider\",\"methods\":\"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows\",\"release\":\"2.7.8\",\"deprecated\":\"false\",\"dubbo\":\"2.0.2\",\"threads\":\"500\",\"pid\":\"56132\",\"interface\":\"a.b.c.bd.api.IWorkflowWebService\",\"version\":\"5.0\",\"generic\":\"false\",\"timeout\":\"30000\",\"revi
sion\":\"5.3-MASTER-20220608.095739-141\",\"path\":\"WorkflowWebService\",\"protocol\":\"dubbo\",\"metadata-type\":\"remote\",\"application\":\"dig-buw-dispatcher-app\",\"payload\":\"104857600\",\"dynamic\":\"true\",\"category\":\"providers\",\"anyhost\":\"true\",\"group\":\"public\",\"timestamp\":\"1655988109818\"},\"enabled\":true,\"weight\":1.0,\"clusterName\":\"DEFAULT\",\"serviceName\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"ephemeral\":true}],\"dom\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"name\":\"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public\",\"cacheMillis\":10000,\"lastRefTime\":1666146440499,\"checksum\":\"aef2359632324193f6bbced5589e6846\",\"useSpecifiedURL\":false,\"clusters\":\"\",\"env\":\"\",\"metadata\":{}}","lastRefTime":56135284087143858} from /10.177.56.132
2022-10-19 10:27:20.859 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] removed ips(1) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public -> [{"instanceId":"10.177.56.133#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.133","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,startWorkflow,startScheduleWorkflow,executeTaskForTemp,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkfl
ow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"2378","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987866761"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 10:27:20.860 INFO [com.alibaba.nacos.naming.push.receiver:client.naming] current ips:(3) service: DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public -> [{"instanceId":"10.177.56.132#20882#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.132","port":20882,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkf
low,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"53008","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987760900"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.134#20881#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.134","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,re
sumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"31262","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655987989873"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000},{"instanceId":"10.177.56.135#2088
1#DEFAULT#DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","ip":"10.177.56.135","port":20881,"weight":1.0,"healthy":true,"enabled":true,"ephemeral":true,"clusterName":"DEFAULT","serviceName":"DEFAULT_GROUP@@providers:a.b.c.bd.api.IWorkflowWebService:5.0:public","metadata":{"side":"provider","methods":"startLaunchedWorkflowByCode,restartWorkflows,restartWorkflowNew,resumeWorkflows,restartWorkflow,recoverWorkflowNew,resumeWorkflow,startLaunchedWorkflow,dealWithCacheRestart,suspendWorkflow,suspendWorkflows,reRunWorkflow,returnAuditTaskResult,dealWithCacheCrash,recoverWorkflows,executeTaskForTemp,startWorkflow,startScheduleWorkflow,returnCheckResult,doNodeStopForKill,returnTaskResult,getParamDefaultValue,killWarnWorkflows,recoverWorkflow,dealParentWorkflow,killWorkflows","release":"2.7.8","deprecated":"false","dubbo":"2.0.2","threads":"500","pid":"56132","interface":"a.b.c.bd.api.IWorkflowWebService","version":"5.0","generic":"false","timeout":"30000","revision":"5.
3-MASTER-20220608.095739-141","path":"WorkflowWebService","protocol":"dubbo","metadata-type":"remote","application":"dig-buw-dispatcher-app","payload":"104857600","dynamic":"true","category":"providers","anyhost":"true","group":"public","timestamp":"1655988109818"},"ipDeleteTimeout":30000,"instanceHeartBeatInterval":5000,"instanceHeartBeatTimeOut":15000}]
2022-10-19 11:36:36.582 WARN [com.alibaba.nacos.client.naming.updater:client.naming] failed to request http://10.177.56.130:9908/nacos/v1/ns/instance/list?app=unknown&healthyOnly=false&namespaceId=10_177_37_101_hh_process_cjzh_src_realtime&clientIP=10.177.56.132&serviceName=DEFAULT_GROUP%40%40providers%3Acom.bonc.dataplatform.buw.executor.api.ITaskService%3A5.0%3AshellScript&udpPort=51084&encoding=UTF-8 from 10.177.56.130
2022-10-19 11:36:36.583 WARN [com.alibaba.nacos.client.naming.updater:client.naming] failed to request http://10.177.56.130:9908/nacos/v1/ns/instance/list?app=unknown&healthyOnly=false&namespaceId=10_177_37_111_hh_process_cjzh_src_realtime&clientIP=10.177.56.132&serviceName=DEFAULT_GROUP%40%40providers%3Acom.bonc.dataplatform.buw.executor.api.ITaskService%3A5.0%3AshellScript&udpPort=48579&encoding=UTF-8 from 10.177.56.130
2022-10-19 11:36:36.583 ERROR [com.alibaba.nacos.client.naming.updater:client.naming] [NA] failed to request
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_181]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_181]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_181]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:1.8.0_181]
at com.alibaba.nacos.client.naming.net.HttpClient.request(HttpClient.java:86) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.callServer(NamingProxy.java:432) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:467) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:400) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:396) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.queryList(NamingProxy.java:321) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.core.HostReactor.updateServiceNow(HostReactor.java:282) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.core.HostReactor$UpdateTask.run(HostReactor.java:330) ~[nacos-client-1.3.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
2022-10-19 11:36:36.584 ERROR [com.alibaba.nacos.client.naming.updater:client.naming] [NA] failed to request
java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_181]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_181]
at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_181]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[?:1.8.0_181]
at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[?:1.8.0_181]
at com.alibaba.nacos.client.naming.net.HttpClient.request(HttpClient.java:86) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.callServer(NamingProxy.java:432) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:467) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:400) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.reqAPI(NamingProxy.java:396) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.net.NamingProxy.queryList(NamingProxy.java:321) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.core.HostReactor.updateServiceNow(HostReactor.java:282) ~[nacos-client-1.3.0.jar:?]
at com.alibaba.nacos.client.naming.core.HostReactor$UpdateTask.run(HostReactor.java:330) ~[nacos-client-1.3.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1323239985
可以到 https://github.com/alibaba/nacos/issues 提个 issue 看下是什么问题，这个日志看起来是 nacos 内部同步用的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10965: nacos+dubbo服务负载，某消费端标记提供者节点为isAvailable导致负载不均衡
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10965:
URL: https://github.com/apache/dubbo/issues/10965#issuecomment-1321388447
> 出现异常状态时，A节点的日志存在【Consumer is shutting down】，关联的https://github.com/apache/dubbo/issues/6294
这个的完整报错堆栈可以贴一下吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org