You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by GitBox <gi...@apache.org> on 2022/10/11 05:53:57 UTC
[GitHub] [dubbo] neverlaughtingboy opened a new issue, #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
neverlaughtingboy opened a new issue, #10733:
URL: https://github.com/apache/dubbo/issues/10733
这是具体报错的堆栈信息，这种错误有没有办法避免
2022-10-10 17:28:42.890 [RID:] [traceId:] [nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent] ERROR c.a.nacos.common.notify.NotifyCenter [DefaultPublisher.java:210] Event callback exception:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3bb61a6e rejected from java.util.concurrent.ScheduledThreadPoolExecutor@52260039[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 31]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
at org.apache.dubbo.registry.RegistryNotifier.notify(RegistryNotifier.java:77)
at org.apache.dubbo.registry.nacos.NacosRegistry$RegistryChildListenerImpl.onEvent(NacosRegistry.java:715)
at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:124)
at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:40)
at com.alibaba.nacos.common.notify.DefaultPublisher.lambda$notifySubscriber$0(DefaultPublisher.java:201)
at com.alibaba.nacos.common.notify.DefaultPublisher.notifySubscriber(DefaultPublisher.java:208)
at com.alibaba.nacos.common.notify.DefaultPublisher.receiveEvent(DefaultPublisher.java:192)
at com.alibaba.nacos.common.notify.DefaultPublisher.openEventHandler(DefaultPublisher.java:118)
at com.alibaba.nacos.common.notify.DefaultPublisher.run(DefaultPublisher.java:95)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1278392929
> > 这个问题复现频率有多高，看起来是 Dubbo 已经反订阅后销毁了线程池了，Nacos 还是推送了数据
>
> 频率的话蛮高的，4个pod滚动部署有2个pod会出现这个报错，理论上 offline 指令发送后应该不会推送过来才对，毕竟我看到 dubbo 入口的流量已经没有了
这个在本地能搭个复现的场景嘛
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] neverlaughtingboy commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
neverlaughtingboy commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1278514452
> > > 这个问题复现频率有多高，看起来是 Dubbo 已经反订阅后销毁了线程池了，Nacos 还是推送了数据
> >
> >
> > 频率的话蛮高的，4个pod滚动部署有2个pod会出现这个报错，理论上 offline 指令发送后应该不会推送过来才对，毕竟我看到 dubbo 入口的流量已经没有了
>
> 这个在本地能搭个复现的场景嘛
本地的话不太好模拟滚动部署，还有没有别的方式或者你这边还需要啥额外的信息
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1275518793
这个报错是在 offline 的过程中出现的还是进程 shutdown 的过程中出现的。另外使用的是 qos 的下线方式是通过 `offline` 命令还是 `shutdown` 命令。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1278532738
> > > > 这个问题复现频率有多高，看起来是 Dubbo 已经反订阅后销毁了线程池了，Nacos 还是推送了数据
> > >
> > >
> > > 频率的话蛮高的，4个pod滚动部署有2个pod会出现这个报错，理论上 offline 指令发送后应该不会推送过来才对，毕竟我看到 dubbo 入口的流量已经没有了
> >
> >
> > 这个在本地能搭个复现的场景嘛
>
> 本地的话不太好模拟滚动部署，还有没有别的方式或者你这边还需要啥额外的信息
能不能把从 shutdownhook 开始触发的到结束的所有日志发一下，数据多的话也可以直接发我邮箱
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] neverlaughtingboy commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
neverlaughtingboy commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1276976896
> 这个问题复现频率有多高，看起来是 Dubbo 已经反订阅后销毁了线程池了，Nacos 还是推送了数据
频率的话蛮高的，4个pod滚动部署有2个pod会出现这个报错，理论上 offline 指令发送后应该不会推送过来才对，毕竟我看到 dubbo 入口的流量已经没有了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] neverlaughtingboy commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
neverlaughtingboy commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1278658565
> > > > > 这个问题复现频率有多高，看起来是 Dubbo 已经反订阅后销毁了线程池了，Nacos 还是推送了数据
> > > >
> > > >
> > > > 频率的话蛮高的，4个pod滚动部署有2个pod会出现这个报错，理论上 offline 指令发送后应该不会推送过来才对，毕竟我看到 dubbo 入口的流量已经没有了
> > >
> > >
> > > 这个在本地能搭个复现的场景嘛
> >
> >
> > 本地的话不太好模拟滚动部署，还有没有别的方式或者你这边还需要啥额外的信息
>
> 能不能把从 shutdownhook 开始触发的到结束的所有日志发一下，数据多的话也可以直接发我邮箱
已经发送，请查收，部分敏感信息手动做了处理
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
AlbumenJ commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1276954206
这个问题复现频率有多高，看起来是 Dubbo 已经反订阅后销毁了线程池了，Nacos 还是推送了数据
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] neverlaughtingboy commented on issue #10733: dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
Posted by GitBox <gi...@apache.org>.
neverlaughtingboy commented on issue #10733:
URL: https://github.com/apache/dubbo/issues/10733#issuecomment-1276042162
> 这个报错是在 offline 的过程中出现的还是进程 shutdown 的过程中出现的。另外使用的是 qos 的下线方式是通过 `offline` 命令还是 `shutdown` 命令。
用的 offline 命令，应该是在执行 SpringApplicationShutdownHook 过程中出现的，补充一下上下文日志
2022-10-10 17:28:26.145 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO o.a.d.c.r.GlobalResourcesRepository [GlobalResourcesRepository.java:120] [DUBBO] Dubbo is completely destroyed, dubbo version: 3.0.11, current host: 172.xx.xxx.xx
2022-10-10 17:28:26.146 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO o.a.d.c.t.m.FrameworkExecutorRepository [FrameworkExecutorRepository.java:175] [DUBBO] destroying framework executor repository .., dubbo version: 3.0.11, current host: 172.xx.xxx.xx
2022-10-10 17:28:26.147 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO o.a.d.c.s.c.DubboSpringInitializer [DubboSpringInitializer.java:78] [DUBBO] Unbind Dubbo Module[1.1.1] from spring container: org.springframework.beans.factory.support.DefaultListableBeanFactory@5c83ae01, dubbo version: 3.0.11, current host: 172.xx.xxx.xx
[INFO] [10/10/2022 17:28:26.251] [172018251086_1_79792-akka.remote.remote-dispatcher-thread8-88] [akka.tcp://172018251086_1_79792@172.18.251.86:40127/system/remoting-terminator] Shutting down remote daemon.
[INFO] [10/10/2022 17:28:26.254] [172018251086_1_79792-akka.remote.remote-dispatcher-thread8-88] [akka.tcp://172018251086_1_79792@172.18.251.86:40127/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports.
[INFO] [10/10/2022 17:28:26.302] [172018251086_1_79792-akka.actor.default-dispatcher-303] [akka.remote.Remoting] Remoting shut down
[INFO] [10/10/2022 17:28:26.304] [172018251086_1_79792-akka.remote.remote-dispatcher-thread8-16] [akka.tcp://172018251086_1_79792@172.18.251.86:40127/system/remoting-terminator] Remoting shut down.
2022-10-10 17:28:42.890 [RID:] [traceId:] [nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent] ERROR c.a.nacos.common.notify.NotifyCenter [DefaultPublisher.java:210] Event callback exception:
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@197de0ca rejected from java.util.concurrent.ScheduledThreadPoolExecutor@5ba4f606[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 21]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632)
at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
at org.apache.dubbo.registry.RegistryNotifier.notify(RegistryNotifier.java:77)
at org.apache.dubbo.registry.nacos.NacosRegistry$RegistryChildListenerImpl.onEvent(NacosRegistry.java:715)
at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:124)
at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:40)
at com.alibaba.nacos.common.notify.DefaultPublisher.lambda$notifySubscriber$0(DefaultPublisher.java:201)
at com.alibaba.nacos.common.notify.DefaultPublisher.notifySubscriber(DefaultPublisher.java:208)
at com.alibaba.nacos.common.notify.DefaultPublisher.receiveEvent(DefaultPublisher.java:192)
at com.alibaba.nacos.common.notify.DefaultPublisher.openEventHandler(DefaultPublisher.java:118)
at com.alibaba.nacos.common.notify.DefaultPublisher.run(DefaultPublisher.java:95)
2022-10-10 17:28:58.314 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO c.b.d.d.DynamicRoutingDataSource [DynamicRoutingDataSource.java:205] dynamic-datasource start closing ....
2022-10-10 17:28:58.318 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO c.alibaba.druid.pool.DruidDataSource [DruidDataSource.java:2029] {dataSource-3} closing ...
2022-10-10 17:28:58.321 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO c.alibaba.druid.pool.DruidDataSource [DruidDataSource.java:2101] {dataSource-3} closed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org