You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2022/11/22 07:36:51 UTC
[GitHub] [skywalking] Yelijah opened a new issue, #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Yelijah opened a new issue, #10001:
URL: https://github.com/apache/skywalking/issues/10001
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
### Apache SkyWalking Component
Java Agent (apache/skywalking-java)
### What happened
Deadlock happened when start tomcat with skywalking agent. But it just happened once, then never happened after that time.
here is jstack :
```
Found one Java-level deadlock:
=============================
"SkywalkingAgent-12-GRPCChannelManager-0":
waiting to lock monitor 0x00005624750588a8 (object 0x00000005800666a0, a java.util.jar.JarFile),
which is held by "main"
"main":
waiting to lock monitor 0x0000562475f7a308 (object 0x000000059d482db0, a java.lang.Object),
which is held by "FileHandlerLogFilesCleaner-1"
"FileHandlerLogFilesCleaner-1":
waiting to lock monitor 0x00005624750588a8 (object 0x00000005800666a0, a java.util.jar.JarFile),
which is held by "main"
Java stack information for the threads listed above:
===================================================
"SkywalkingAgent-12-GRPCChannelManager-0":
at java.util.zip.ZipFile.getEntry(ZipFile.java:314)
- waiting to lock <0x00000005800666a0> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getEntry(JarFile.java:241)
at java.util.jar.JarFile.getJarEntry(JarFile.java:224)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1056)
at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
- locked <0x00000005d671fdb8> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.skywalking.apm.dependencies.io.grpc.Metadata.<clinit>(Metadata.java:114)
at org.apache.skywalking.apm.agent.core.remote.AgentIDDecorator.<clinit>(AgentIDDecorator.java:43)
at org.apache.skywalking.apm.agent.core.remote.GRPCChannelManager.run(GRPCChannelManager.java:145)
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:33)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"main":
at java.lang.ClassLoader.loadClass(ClassLoader.java:400)
- waiting to lock <0x000000059d482db0> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType$TokenList.<init>(TypePool.java:6904)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$ForType$Tokenized.resolveInterfaceTypes(TypePool.java:4078)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription.getInterfaces(TypePool.java:2780)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfParameterizedType.getInterfaces(TypeDescription.java:4927)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$LazyProjection$WithEagerNavigation.getInterfaces(TypeDescription.java:6502)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:67)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:32)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:146)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
at org.apache.skywalking.apm.agent.core.plugin.match.ProtectiveShieldMatcher.matches(ProtectiveShieldMatcher.java:47)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1833)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11880)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11838)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11555)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12238)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12178)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11747)
at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
at sun.security.util.DerOutputStream.<clinit>(DerOutputStream.java:359)
at sun.security.util.NamedCurve.<init>(NamedCurve.java:60)
at sun.security.util.CurveDB.add(CurveDB.java:138)
at sun.security.util.CurveDB.<clinit>(CurveDB.java:181)
at sun.security.util.DisabledAlgorithmConstraints$Constraints.<init>(DisabledAlgorithmConstraints.java:311)
at sun.security.util.DisabledAlgorithmConstraints.<init>(DisabledAlgorithmConstraints.java:128)
at sun.security.util.DisabledAlgorithmConstraints.<init>(DisabledAlgorithmConstraints.java:95)
at sun.security.util.SignatureFileVerifier.<clinit>(SignatureFileVerifier.java:62)
at java.util.jar.JarFile.initializeVerifier(JarFile.java:371)
at java.util.jar.JarFile.ensureInitialization(JarFile.java:615)
- locked <0x00000005800666a0> (a java.util.jar.JarFile)
at java.util.jar.JavaUtilJarAccessImpl.ensureInitialization(JavaUtilJarAccessImpl.java:69)
at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:993)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:451)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
- locked <0x000000058db3d4a0> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:601)
"FileHandlerLogFilesCleaner-1":
at java.util.zip.ZipFile.getEntry(ZipFile.java:314)
- waiting to lock <0x00000005800666a0> (a java.util.jar.JarFile)
at java.util.jar.JarFile.getEntry(JarFile.java:241)
at java.util.jar.JarFile.getJarEntry(JarFile.java:224)
at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1056)
at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
- locked <0x000000059d482db0> (a java.lang.Object)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType$TokenList.<init>(TypePool.java:6904)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$GenericTypeToken$Resolution$ForType$Tokenized.resolveInterfaceTypes(TypePool.java:4078)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription.getInterfaces(TypePool.java:2780)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$OfParameterizedType.getInterfaces(TypeDescription.java:4927)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.description.type.TypeDescription$Generic$LazyProjection$WithEagerNavigation.getInterfaces(TypeDescription.java:6502)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:67)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.HasSuperTypeMatcher.doMatch(HasSuperTypeMatcher.java:32)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$ForNonNullValues.matches(ElementMatcher.java:249)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Conjunction.matches(ElementMatcher.java:146)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.matcher.ElementMatcher$Junction$Disjunction.matches(ElementMatcher.java:214)
at org.apache.skywalking.apm.agent.core.plugin.match.ProtectiveShieldMatcher.matches(ProtectiveShieldMatcher.java:47)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$RawMatcher$ForElementMatchers.matches(AgentBuilder.java:1833)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doTransform(AgentBuilder.java:11880)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11838)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.access$1700(AgentBuilder.java:11555)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12238)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer$LegacyVmDispatcher.run(AgentBuilder.java:12178)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.doPrivileged(AgentBuilder.java)
at org.apache.skywalking.apm.dependencies.net.bytebuddy.agent.builder.AgentBuilder$Default$ExecutingTransformer.transform(AgentBuilder.java:11747)
at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
at java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem(FileSystems.java:95)
at java.nio.file.FileSystems$DefaultFileSystemHolder.<clinit>(FileSystems.java:90)
at java.nio.file.FileSystems.getDefault(FileSystems.java:176)
at java.io.File.toPath(File.java:2234)
- locked <0x000000059d453680> (a java.io.File)
at org.apache.juli.FileHandler.streamFilesForDelete(FileHandler.java:557)
at org.apache.juli.FileHandler.lambda$clean$0(FileHandler.java:544)
at org.apache.juli.FileHandler$$Lambda$17/439925336.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Found 1 deadlock.
```
### What you expected to happen
The deadlock never happened when use tomcat with skywalking agent.
### How to reproduce
it happened only once.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324497373
Even with this single stack, you can see the Apache Juli holding the file and the agent class loader trying to lock.
How could this happen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323262616
The agent with jars(core and plugins), config files and logs should be put out of the whole Tomcat folder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324473077
> But the stack says so, there is nothing I can do. This never happens on my side or in anyone's env I knew. I can only provide helps based on what you provided. You can't say this is not possible but JVM confirmed yes.
<img width="1107" alt="image" src="https://user-images.githubusercontent.com/33784573/203456716-48394c4c-ff3c-48a8-bbdc-2ca1a5731740.png">
The stack told me it's nothing about file delete lock (0x000000059d453680), the deadlock is 0x000000059d482db0 and 0x00000005800666a0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324506346
I think the dead lock progress is :
1. main thread try to load a class in one jar(alias it as jar A) then lock this jar (0x00000005800666a0, hold by JarFile.ensureInitialization), then trigger skywalking tranfromer, then skywalking.net.bytebuddy.pool.TypePool.TokenList init, then trigger some class(alias it as class B) begin load and wait this class lock.
2. FileHandlerLogFilesCleaner thread used FileSystems.getDefault method which triggered skywalking tranfromer, then skywalking.net.bytebuddy.pool.TypePool.TokenList init, then trigger some class(alias it as class B) begin load and get class lock (0x000000059d482db0, held by ClassLoader.loadClass), then URLClassLoader try find class by scan all jars, but jarA is locked by main.
FileHandlerLogFilesCleaner is async , so this deadlock is quite rare
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323254165
But, why it going to remove jar files? Where do you put plugin agent.jar and plugin jars? They should out of Tomcat's control.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1325119187
> In our case, we can't. If you ignore that, the bootstrap plugin is not working, I think.
Yes, it's the risk of using bootstrap plugins. I din't need them, so I prepare to custom skywalking source code in my case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1325108877
In our case, we can't. If you ignore that, the bootstrap plugin is not working, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324444194
> The agent with jars(core and plugins), config files and logs should be put out of the whole Tomcat folder.
I'm sorry, I didn't put agent in tomcat webapps/xxx/WEB-INF at all but in /home/skywalking-agent. So I'm quite sure it can't tomcat's FileHandlerLogFilesCleaner didn't try to delete any agent's file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323294267
> Check this
>
> ```
> "SkywalkingAgent-12-GRPCChannelManager-0":
> waiting to lock monitor 0x00005624750588a8 (object 0x00000005800666a0, a java.util.jar.JarFile),
> which is held by "main"
> "main":
> waiting to lock monitor 0x0000562475f7a308 (object 0x000000059d482db0, a java.lang.Object),
> which is held by "FileHandlerLogFilesCleaner-1"
> "FileHandlerLogFilesCleaner-1":
> waiting to lock monitor 0x00005624750588a8 (object 0x00000005800666a0, a java.util.jar.JarFile),
> which is held by "main"
> ```
>
> 0x00005624750588a8 is held by both FileHandlerLogFilesCleaner and main/agent.
I see that. But the jar file's lock is triggered by classloader.findclass, and the classloader is both triggered by main thread and FileHandlerLogFilesCleaner thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323253163
> It is new to me. What is `FileHandlerLogFilesCleaner` in your env? Why it going to touch jar lib? This seems unexpected.
FileHandlerLogFilesCleaner seems to be tomcat's log clean module which defines in tomcat's conf/logging.properties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324516369
> > FileSystems.getDefault method which triggered skywalking tranfromer
>
> Why does this method trigger that? FileSystems is from the bootstrap classloader, SkyWalking doesn't instrument even watch its loading.
I'm confused too, but the stack sayed so
<img width="1041" alt="image" src="https://user-images.githubusercontent.com/33784573/203466286-61ddd145-420c-4a0e-9680-7efdcac3b5c8.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324579405
I am not sure actually. Plugins are contributed by many people. I don't know all the details. I could only say, we don't instrument any Bootstrap Class in the agent core.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324959411
> I am not sure actually. Plugins are contributed by many people. I don't know all the details. I could only say, we don't instrument any Bootstrap Class in the agent core.
I know what's wrong, `SkyWalkingAgent` set ignore for `AgentBuilder`, this will override bytebuddy's default ignore which ignore all classes loaded by BootstrapClassLoader.
So i have to add one more ignore for all classes loaded by BootstrapClassLoader
```java
public Default(ByteBuddy byteBuddy) {
this(byteBuddy,
Listener.NoOp.INSTANCE,
DEFAULT_LOCK,
PoolStrategy.Default.FAST,
TypeStrategy.Default.REBASE,
LocationStrategy.ForClassLoader.STRONG,
NativeMethodStrategy.Disabled.INSTANCE,
WarmupStrategy.NoOp.INSTANCE,
TransformerDecorator.NoOp.INSTANCE,
new InitializationStrategy.SelfInjection.Split(),
RedefinitionStrategy.DISABLED,
RedefinitionStrategy.DiscoveryStrategy.SinglePass.INSTANCE,
RedefinitionStrategy.BatchAllocator.ForTotal.INSTANCE,
RedefinitionStrategy.Listener.NoOp.INSTANCE,
RedefinitionStrategy.ResubmissionStrategy.Disabled.INSTANCE,
InjectionStrategy.UsingReflection.INSTANCE,
LambdaInstrumentationStrategy.DISABLED,
DescriptionStrategy.Default.HYBRID,
FallbackStrategy.ByThrowableType.ofOptionalTypes(),
ClassFileBufferStrategy.Default.RETAINING,
InstallationListener.NoOp.INSTANCE,
new RawMatcher.Disjunction(
new RawMatcher.ForElementMatchers(any(), isBootstrapClassLoader().or(isExtensionClassLoader())),
new RawMatcher.ForElementMatchers(nameStartsWith("net.bytebuddy.")
.and(not(ElementMatchers.nameStartsWith(NamingStrategy.BYTE_BUDDY_RENAME_PACKAGE + ".")))
.or(nameStartsWith("sun.reflect.").or(nameStartsWith("jdk.internal.reflect.")))
.<TypeDescription>or(isSynthetic()))),
Collections.<Transformation>emptyList());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323236503
It is new to me. What is `FileHandlerLogFilesCleaner` in your env? Why it going to touch jar lib? This seems unexpected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323290812
SkyWalking agent even doesn't manipulate the jar file, we just read it. So it is impossible we trigger a deadlock. Only read/remove could be a deadlock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323260386
> But, why it going to remove jar files? Where do you put plugin agent.jar and plugin jars? They should out of Tomcat's control.
I put it in webapps/xxx/WEB-INF/skywalking-java-agent. I think the FileHandlerLogFilesCleaner thread tried to move log in tomcat's logs dir but not app's dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324518072
I see this. I don't know what causes this. I think if you want to dig deeper, you should ask byte-buddy community for helping.
The author knows much more about the instrumentation details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324533766
I put apm-jdk-threading-plugin-8.12.0.jar into plugins, does it effect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1325120124
Thank you for your time.
> In our case, we can't. If you ignore that, the bootstrap plugin is not working, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] Yelijah commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
Yelijah commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323281868
> The agent with jars(core and plugins), config files and logs should be put out of the whole Tomcat folder.
I don't think
> But, why it going to remove jar files? Where do you put plugin agent.jar and plugin jars? They should out of Tomcat's control.
I don't think FileHandlerLogFilesCleaner is trying to delete jar. The deadlock was caused by classloader
<img width="1028" alt="image" src="https://user-images.githubusercontent.com/33784573/203262259-aaa87128-2ba7-4bba-a96c-e6d62eac9252.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
URL: https://github.com/apache/skywalking/issues/10001
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323287892
Check this
```
"SkywalkingAgent-12-GRPCChannelManager-0":
waiting to lock monitor 0x00005624750588a8 (object 0x00000005800666a0, a java.util.jar.JarFile),
which is held by "main"
"main":
waiting to lock monitor 0x0000562475f7a308 (object 0x000000059d482db0, a java.lang.Object),
which is held by "FileHandlerLogFilesCleaner-1"
"FileHandlerLogFilesCleaner-1":
waiting to lock monitor 0x00005624750588a8 (object 0x00000005800666a0, a java.util.jar.JarFile),
which is held by "main"
```
0x00005624750588a8 is held by both FileHandlerLogFilesCleaner and main/agent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1323299451
Of course, when a classloader is reading this file, and another thread is trying to delete that, then deadlock is triggered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324445183
But the stack says so, there is nothing I can do. This never happens on my side or in anyone's env I knew.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #10001: [Bug] Deadlock happened when start tomcat with skywalking agent
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #10001:
URL: https://github.com/apache/skywalking/issues/10001#issuecomment-1324514630
> FileSystems.getDefault method which triggered skywalking tranfromer
Why does this method trigger that? File is a bootstrap classloader, SkyWalking doesn't instrument even watch its loading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org