You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2019/11/05 02:50:18 UTC
[GitHub] [skywalking] zhaoyuguang opened a new issue #3784: Project deadlock
when starting application
zhaoyuguang opened a new issue #3784: Project deadlock when starting application
URL: https://github.com/apache/skywalking/issues/3784
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [x] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement
___
### Question
- What do you want to know?
Project deadlock discussion
#### Found one Java-level deadlock:
"catalina-exec-311":
waiting for ownable synchronizer 0x00000005ed54a980, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "catalina-exec-57"
"catalina-exec-57":
waiting to lock monitor 0x00007efa5c0088a8 (object 0x00000005ed47db58, a org.apache.catalina.loader.WebappClassLoader),
which is held by "catalina-exec-311"
#### "catalina-exec-311":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000005ed54a980> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:70)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.<init>(InstMethodsInter.java:56)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhanceInstance(ClassEnhancePluginDefine.java:162)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.ClassEnhancePluginDefine.enhance(ClassEnhancePluginDefine.java:74)
at org.apache.skywalking.apm.agent.core.plugin.AbstractClassEnhancePluginDefine.define(AbstractClassEnhancePluginDefine.java:75)
at org.apache.skywalking.apm.agent.SkyWalkingAgent$Transformer.transform(SkyWalkingAgent.java:145)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Transformer$Compound.transform(AgentBuilder.java:2335)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$Transformation$Simple$Resolution.apply(AgentBuilder.java:8902)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:9306)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9269)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1300(AgentBuilder.java:9047)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9625)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:9575)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:9194)
at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:3148)
- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:1388)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1876)
- locked <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1750)
at org.springframework.util.ClassUtils.forName(ClassUtils.java:250)
at org.springframework.http.client.HttpComponentsClientHttpRequestFactory.<clinit>(HttpComponentsClientHttpRequestFactory.java:67)
at com.xyz.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:xx)
at com.xyz.RestTemplateFactoryBean.getObject(RestTemplateFactoryBean.java:xx)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
- locked <0x00000005f8602888> (a java.util.concurrent.ConcurrentHashMap)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1634)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1081)
#### "catalina-exec-57":
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1784)
- waiting to lock <0x00000005ed47db58> (a org.apache.catalina.loader.WebappClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
- locked <0x00000007a5a655e0> (a java.lang.Object)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.skywalking.apm.agent.core.plugin.loader.InterceptorInstanceLoader.load(InterceptorInstanceLoader.java:77)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.StaticMethodsInter.intercept(StaticMethodsInter.java:73)
### My analysis
First, `catalina-exec-311` and `catalina-exec-57` have a deadlock at the following location;
I think SkyWalking is trying to make all the enhanced classes,
First get the `ReentrantLock INSTANCE_LOAD_LOCK` lock of sw,
Then classload.
But thread 311 destroys this rule, see this code
at org.springframework.http.client.HttpComponentsClientHttpRequestFactory.<clinit>(HttpComponentsClientHttpRequestFactory.java:67)
This first get classload lock, then the `ReentrantLock INSTANCE_LOAD_LOCK` lock of sw.
So, Deadlock in the case of concurrency. Am I right? and how avoid this case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services