You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dubbo.apache.org by "gulang12 (via GitHub)" <gi...@apache.org> on 2023/06/29 07:01:25 UTC
[GitHub] [dubbo] gulang12 opened a new issue, #12620: 升级dubbo3遇到问题，项目里使用大量的@Scheduled，启动是出现线程死锁导致启动服务失败
gulang12 opened a new issue, #12620:
URL: https://github.com/apache/dubbo/issues/12620
升级dubbo3遇到问题，项目里使用大量的@Scheduled（并且定时任务里RPC调用别的服务），启动是出现线程死锁导致启动服务失败
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] tang2992 commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "tang2992 (via GitHub)" <gi...@apache.org>.
tang2992 commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1615534353
@PostConstruct和@scheduled都是spring的常规代码，有办法解决避免这个死锁吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1634151148
> > Spring 启动有几个阶段
> >
> > 1. 加载全部的 definition -> Dubbo 在这个时间获取全量的配置
> > 2. 加载 ApplicationListener -> Dubbo 的 Listener 初始化
> > 3. 发布 EarlyEvent -> Dubbo 在这个时候初始化配置
> >
> > 如果在阶段 2 发起了 Dubbo 调用会存在死锁
> > 这个修改的原因是如果在阶段 1 直接初始化了所有配置，很可能存在：
> >
> > 1. Registry、Application 等配置没有全量加载，部分服务发布、订阅非预期
> > 2. Reference 的服务早于 Service 的服务初始化，导致本地循环依赖的服务启动 check 失败
>
> 这个在3.1.5具体怎么解决，有解决方案吗，因为我们之前调研版本的时候在官网看到3.1.5是稳定版本了，所以现在所有应用都在升级3.1.5版本，这个问题不解决没法升级
对于子任务中使用 Dubbo 调用的等待应用完全启动后再发起调用
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] tang2992 commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "tang2992 (via GitHub)" <gi...@apache.org>.
tang2992 commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1615336931
我也遇到了这个问题
Found one Java-level deadlock:
=============================
"hsTaskScheduler-1":
waiting to lock monitor 0x00000242c43b94a8 (object 0x00000006c4b96140, a org.apache.dubbo.config.deploy.DefaultModuleDeployer),
which is held by "main"
"main":
waiting to lock monitor 0x00000242c583e248 (object 0x00000006c3a75060, a java.util.concurrent.ConcurrentHashMap),
which is held by "hsTaskScheduler-1"
Java stack information for the threads listed above:
===================================================
"hsTaskScheduler-1":
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.startSync(DefaultModuleDeployer.java:143)
- waiting to lock <0x00000006c4b96140> (a org.apache.dubbo.config.deploy.DefaultModuleDeployer)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.start(DefaultModuleDeployer.java:139)
at org.apache.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:228)
at org.apache.dubbo.config.spring.ReferenceBean.getCallProxy(ReferenceBean.java:351)
- locked <0x00000006c3a75060> (a java.util.concurrent.ConcurrentHashMap)
at org.apache.dubbo.config.spring.ReferenceBean.access$100(ReferenceBean.java:100)
at org.apache.dubbo.config.spring.ReferenceBean$DubboReferenceLazyInitTargetSource.createObject(ReferenceBean.java:359)
at org.springframework.aop.target.AbstractLazyCreationTargetSource.getTarget(AbstractLazyCreationTargetSource.java:86)
- locked <0x00000006c61d99e8> (a org.apache.dubbo.config.spring.ReferenceBean$DubboReferenceLazyInitTargetSource)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:192)
at com.sun.proxy.$Proxy105.query(Unknown Source)
at com.huasheng.platform.client.service.app.config.AppConfigReadServiceClient.query(AppConfigReadServiceClient.java:57)
at com.huasheng.stock.search.task.LoadStockDefinitionTaskV2.execute(LoadStockDefinitionTaskV2.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
"main":
at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:187)
- waiting to lock <0x00000006c3a75060> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:128)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.apache.dubbo.config.spring.ServiceBean.publishExportEvent(ServiceBean.java:133)
at org.apache.dubbo.config.spring.ServiceBean.exported(ServiceBean.java:125)
at org.apache.dubbo.config.ServiceConfig.doExport(ServiceConfig.java:392)
- locked <0x000000078da98a88> (a org.apache.dubbo.config.spring.ServiceBean)
at org.apache.dubbo.config.ServiceConfig.export(ServiceConfig.java:243)
- locked <0x000000078da98a88> (a org.apache.dubbo.config.spring.ServiceBean)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.exportServiceInternal(DefaultModuleDeployer.java:350)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.exportServices(DefaultModuleDeployer.java:322)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.startSync(DefaultModuleDeployer.java:158)
- locked <0x00000006c4b96140> (a org.apache.dubbo.config.deploy.DefaultModuleDeployer)
at org.apache.dubbo.config.deploy.DefaultModuleDeployer.start(DefaultModuleDeployer.java:139)
at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onContextRefreshedEvent(DubboDeployApplicationListener.java:113)
at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:102)
at org.apache.dubbo.config.spring.context.DubboDeployApplicationListener.onApplicationEvent(DubboDeployApplicationListener.java:47)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:883)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.finishRefresh(EmbeddedWebApplicationContext.java:146)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545)
- locked <0x00000006c3c8e710> (a java.lang.Object)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:124)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107)
at com.huasheng.sns.server.RobotApplication.main(RobotApplication.java:26)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] gulang12 commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "gulang12 (via GitHub)" <gi...@apache.org>.
gulang12 commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1619838686
> Spring 启动有几个阶段
>
> 1. 加载全部的 definition -> Dubbo 在这个时间获取全量的配置
> 2. 加载 ApplicationListener -> Dubbo 的 Listener 初始化
> 3. 发布 EarlyEvent -> Dubbo 在这个时候初始化配置
>
> 如果在阶段 2 发起了 Dubbo 调用会存在死锁
>
> 这个修改的原因是如果在阶段 1 直接初始化了所有配置，很可能存在：
>
> 1. Registry、Application 等配置没有全量加载，部分服务发布、订阅非预期
> 2. Reference 的服务早于 Service 的服务初始化，导致本地循环依赖的服务启动 check 失败
这个在3.1.5具体怎么解决，有解决方案吗，因为我们之前调研版本的时候在官网看到3.1.5是稳定版本了，所以现在所有应用都在升级3.1.5版本，这个问题不解决没法升级
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1617116907
Spring 启动有几个阶段
1. 加载全部的 definition -> Dubbo 在这个时间获取全量的配置
2. 加载 ApplicationListener -> Dubbo 的 Listener 初始化
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] AlbumenJ closed issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "AlbumenJ (via GitHub)" <gi...@apache.org>.
AlbumenJ closed issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
URL: https://github.com/apache/dubbo/issues/12620
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] liufeiyu1002 commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "liufeiyu1002 (via GitHub)" <gi...@apache.org>.
liufeiyu1002 commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1615866731
> @PostConstruct和@scheduled都是spring的常规代码，有办法解决避免这个死锁吗？
不要在初始化方法里调用 dubbo 接口
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] songxiaosheng commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1614127660
那个锁被哪个线程占用了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] songxiaosheng commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "songxiaosheng (via GitHub)" <gi...@apache.org>.
songxiaosheng commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1615373828
- main线程
- 持有：
- 0x00000006c4b96140 这把锁的类型是DefaultModuleDeployer private synchronized Future startSync()
- 0x000000078da98a88 ServiceBean
- 等待锁：
- 0x00000006c3a75060 ConcurrentHashMap Spring广播器使用到的
hsTaskScheduler线程
- 持有：
- 0x00000006c61d99e8 DubboReferenceLazyInitTargetSource
- 0x00000006c3a75060 ConcurrentHashMap 创建Bean时使用到的
- 等待 0x00000006c4b96140 DefaultModuleDeployer
两把锁 一个线程先拿Spring的Map锁，再拿Dubbo的Deployer锁，一个线程先拿Dubbo的Deployer锁，再拿Spring的Map锁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org
[GitHub] [dubbo] liufeiyu1002 commented on issue #12620: dubbo3.1.5版本BUG！！！，启动是出现线程死锁导致启动服务失败
Posted by "liufeiyu1002 (via GitHub)" <gi...@apache.org>.
liufeiyu1002 commented on issue #12620:
URL: https://github.com/apache/dubbo/issues/12620#issuecomment-1615869671
> @PostConstruct和@scheduled都是spring的常规代码，有办法解决避免这个死锁吗？
从你的堆栈看，大概 应该是 `@PostConstruct` 修饰的初始化方法里 调用了 `@Scheduled` 注解修饰的一个方法，这个方法里 调用了 dubbo 接口 导致的。 目前避免的话 就是 不要在初始化方法里 调用 `dubbo `接口 。 虽然不太清楚 既然用了 `@Scheduled` 为啥还要在初始化再调用一次。。。 不过你可以换个思路解决 比如 监听一下` dubbo application 的 started `事件 再这里调用一次 `@Scheduled` 相关的逻辑
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@dubbo.apache.org
For additional commands, e-mail: notifications-help@dubbo.apache.org