[jira] [Comment Edited] (FELIX-5976) Deadlock can occur when loading class in Permissions.java

["Timothy Carroll (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FELIX-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16673507#comment-16673507 ] 

Timothy Carroll edited comment on FELIX-5976 at 11/2/18 7:26 PM:
-----------------------------------------------------------------

So I'm getting the OK later today to send you more info, but what I can send you now is this which shows that we can't avoid this check with an all permissions. Basically, the problem is the code first checks if the implicit permissions cover the situation. Implicit permissions are never all permissions.
{code:java}
waiting for http-nio-8080-exec-16@37688 to release lock on <0x93ae> (a com.appiancorp.tomcat.loader.AppianWebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1170)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1137)
at org.apache.felix.framework.security.util.Permissions$2.run(Permissions.java:530)
at java.security.AccessController.doPrivileged(AccessController.java:-1)
at org.apache.felix.framework.security.util.Permissions.createPermission(Permissions.java:515)
at org.apache.felix.framework.security.util.Permissions.implies(Permissions.java:336)
at org.apache.felix.framework.security.condpermadmin.ConditionalPermissionAdminImpl.eval(ConditionalPermissionAdminImpl.java:954)
at org.apache.felix.framework.security.condpermadmin.ConditionalPermissionAdminImpl.hasPermission(ConditionalPermissionAdminImpl.java:887)
at org.apache.felix.framework.SecurityProviderImpl.hasBundlePermission(SecurityProviderImpl.java:123)
at org.apache.felix.framework.Felix.impliesBundlePermission(Felix.java:4300)
at org.apache.felix.framework.BundleProtectionDomain.implies(BundleProtectionDomain.java:68)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:450)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.util.zip.ZipFile.<init>(ZipFile.java:216)
at java.util.zip.ZipFile.<init>(ZipFile.java:155)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:130)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:328)
- locked <0x93b0> (a java.lang.Object)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntries(AbstractSingleArchiveResourceSet.java:71){code}
As far as loading the plugin framework in a separate classloader goes, I don't think that would solve the problem. The class it's loading when we get the deadlock is {{java.io.FilePermission}}. Won't trying to load that class always end up delegating to the base class loader?

What's sort of weird to me is that Catalina is holding onto the lock for {{FilePermission}} for so long and trying to get archive entries while it has that lock.
{code:java}
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntries(AbstractSingleArchiveResourceSet.java:66)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:274)
at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281)
at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:218)
at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225)
at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2226)
at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(WebappClassLoaderBase.java:155)
at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(WebappClassLoaderBase.java:144)
at java.security.AccessController.doPrivileged(AccessController.java:-1)
at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:827)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1278)
- locked <0x93ae> (a com.appiancorp.tomcat.loader.AppianWebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1137)
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1464)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1966)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.sun.jersey.core.spi.factory.ResponseImpl.toStatusType(ResponseImpl.java:161)
at com.sun.jersey.core.spi.factory.ResponseBuilderImpl.status(ResponseBuilderImpl.java:132){code}
*EDIT:* In posting this I realized that the lock _was not_ on java.io.FilePermission. Per the stack trace, it's on our own internal class loader that is not parallel enabled. I think that we can actually resolve the issue by simply registering that class loader as parallel enabled.


was (Author: tim.carroll):
So I'm getting the OK later today to send you more info, but what I can send you now is this which shows that we can't avoid this check with an all permissions. Basically, the problem is the code first checks if the implicit permissions cover the situation. Implicit permissions are never all permissions.
{code:java}
waiting for http-nio-8080-exec-16@37688 to release lock on <0x93ae> (a com.appiancorp.tomcat.loader.AppianWebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1170)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1137)
at org.apache.felix.framework.security.util.Permissions$2.run(Permissions.java:530)
at java.security.AccessController.doPrivileged(AccessController.java:-1)
at org.apache.felix.framework.security.util.Permissions.createPermission(Permissions.java:515)
at org.apache.felix.framework.security.util.Permissions.implies(Permissions.java:336)
at org.apache.felix.framework.security.condpermadmin.ConditionalPermissionAdminImpl.eval(ConditionalPermissionAdminImpl.java:954)
at org.apache.felix.framework.security.condpermadmin.ConditionalPermissionAdminImpl.hasPermission(ConditionalPermissionAdminImpl.java:887)
at org.apache.felix.framework.SecurityProviderImpl.hasBundlePermission(SecurityProviderImpl.java:123)
at org.apache.felix.framework.Felix.impliesBundlePermission(Felix.java:4300)
at org.apache.felix.framework.BundleProtectionDomain.implies(BundleProtectionDomain.java:68)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:450)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
at java.util.zip.ZipFile.<init>(ZipFile.java:216)
at java.util.zip.ZipFile.<init>(ZipFile.java:155)
at java.util.jar.JarFile.<init>(JarFile.java:166)
at java.util.jar.JarFile.<init>(JarFile.java:130)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
at org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:181)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.openJarFile(AbstractArchiveResourceSet.java:328)
- locked <0x93b0> (a java.lang.Object)
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntries(AbstractSingleArchiveResourceSet.java:71){code}

As far as loading the plugin framework in a separate classloader goes, I don't think that would solve the problem. The class it's loading when we get the deadlock is {{java.io.FilePermission}}. Won't trying to load that class always end up delegating to the base class loader?

What's sort of weird to me is that Catalina is holding onto the lock for {{FilePermission}} for so long and trying to get archive entries while it has that lock.


{code:java}
at org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntries(AbstractSingleArchiveResourceSet.java:66)
at org.apache.catalina.webresources.AbstractArchiveResourceSet.getResource(AbstractArchiveResourceSet.java:274)
at org.apache.catalina.webresources.StandardRoot.getResourceInternal(StandardRoot.java:281)
at org.apache.catalina.webresources.StandardRoot.getResource(StandardRoot.java:218)
at org.apache.catalina.webresources.StandardRoot.getClassLoaderResource(StandardRoot.java:225)
at org.apache.catalina.loader.WebappClassLoaderBase.findClassInternal(WebappClassLoaderBase.java:2226)
at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(WebappClassLoaderBase.java:155)
at org.apache.catalina.loader.WebappClassLoaderBase$PrivilegedFindClassByName.run(WebappClassLoaderBase.java:144)
at java.security.AccessController.doPrivileged(AccessController.java:-1)
at org.apache.catalina.loader.WebappClassLoaderBase.findClass(WebappClassLoaderBase.java:827)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1278)
- locked <0x93ae> (a com.appiancorp.tomcat.loader.AppianWebappClassLoader)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1137)
at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1464)
at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1966)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.sun.jersey.core.spi.factory.ResponseImpl.toStatusType(ResponseImpl.java:161)
at com.sun.jersey.core.spi.factory.ResponseBuilderImpl.status(ResponseBuilderImpl.java:132){code}

> Deadlock can occur when loading class in Permissions.java
> ---------------------------------------------------------
>
>                 Key: FELIX-5976
>                 URL: https://issues.apache.org/jira/browse/FELIX-5976
>             Project: Felix
>          Issue Type: Bug
>         Environment: Centos, Apache Catalina
>            Reporter: Timothy Carroll
>            Priority: Blocker
>
> My company is attempting to use Felix Framework Security, but we are running into a race condition that can lead to thread deadlock. The Felix Security class causing the deadlock is {{org.apache.felix.framework.security.util.Permissions.java}}
> We're using Felix Security v2.4.0 in a Tomcat Catalina app server v8.5.34. We believe the issue could occur in the latest Felix Security version as well. The line of code in questions is line 530 where the {{Permissions.createPermission}} method attempts to load a class.
> We found that the deadlock occurs if the {{loadClass}} method ends up using the root Catalina class loader, {{WebappClassLoaderBase}}. Essentially, we have a case where we use {{javax.xml}} to serialize or deserialize something in an OSGI bundle which results in a call to {{org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntries}}, which is a largely synchronized method. While attempting to get the entries, we run into a permissions check that needs to obtain the lock managed by {{WebappClassLoaderBase}} because of line 530.
> At the same time, we run into a reversed lock order from some older, pre-existing code that is not directly controlled by us. Essentially, our servlet filters attempt to load a class, obtaining the lock on {{WebappClassLoaderBase}}. The class loader (not in our code) is doing something, however, that needs to load archive entries through OSGI which runs into the lock on {{org.apache.catalina.webresources.AbstractSingleArchiveResourceSet.getArchiveEntries}}. Thus, we have a deadlock.
> Since the pre-existing code that obtains the locks in the order {{WebappClassLoaderBase}} > {{AbstractSingleArchiveResourceSet}} is much more opaque and also preexisting in our codebase, we're hoping that we can figure out how to avoid the check on line 530 of {{Permissions}}. It seems like it could be unnecessary, but we would love some assistance from Karl or anyone else familiar with the security framework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)