You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by Gunnar Brand <Gu...@interface-projects.de> on 2019/09/25 10:46:58 UTC
Performance bottleneck in WebappClassLoaderBase
Hello.
Recently we received complaints from a customer about slow response times (45 seconds) every morning. Every morning starting a fresh session in the browser would require a long wait time for the first user, but once this hurdle had been taken everything seemed to work fine.
We could reproduce this in house, and doing a thread dump during this time span showed the class loader trying to open a jar file. Immediately we suspected a virus scanner slowing access to jar files (and since signatures are updated daily, this happened daily). Blocking the virus scanner from the tomcat folder helped but this is just alleviating the symptoms, not the cure. Doing a full trace log of the class loader package showed the problem:
For any unknown resource or class the class loader seems to populate all known jars, and either finds it, then loads it, and keeps a cache entry (with the class for classes), or if it can’t find it, asks the parent class loader (this is the spec order of things).
The log was filled classes and resources that couldn’t be found and were delegated to the parent. Over and over again. Each and every one of theses requests scans all jar files. Clear the OS cache and the problem appears, too.
For classes one would believe this cannot be, once a class is loaded, it’s loaded. But with dynamic script languages, they often use Class.forName() and for example Rhino even does that for packages (that might not even exist!).
We fixed at least Rhino with an application wide global top level , so for classes this problem did go away (there might be other, unless they implement their own caching, there is a risk).
Resources are still a problem, our log is filled with
24-Sep-2019 17:22:55.482 FINE [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.findResources findResources(META-INF/services/javax.xml.parsers.DocumentBuilderFactory)
24-Sep-2019 17:22:55.484 FINE [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream getResourceAsStream(META-INF/services/org.apache.xerces.xni.parser.XMLParserConfiguration)
24-Sep-2019 17:22:55.484 FINE [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream Searching local repositories
24-Sep-2019 17:22:55.485 FINE [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream Delegating to parent classloader unconditionally java.net.URLClassLoader@578486a3
24-Sep-2019 17:22:55.485 FINE [http-nio-8080-exec-6] org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream --> Resource not found, returning null
If there is only a single repeated “Searching local repositories” for a certain URL, the problem is there (doesn’t matter if the resource or class exists or not).
It’s obviously a bad idea to scan the filesystem or jars every time.
The WebappClassLoaderBase keeps information about every resource and class it finds, so repeated calls for these do not cost that much time, especially for classes that are stored in the cache.
I believe it might work to keep a negative entry for classes it doesn’t find but the parent class loader knows and immediately ask the parent class loader. Maybe also for resources. But I don’t know what is supposed to happen if somebody overwrites a resource (or class) later.
Even worse if the class/resource does not exist yet. Maybe populating the whole resource tree in memory is the only solution (with background update?!) .
I mainly wanted to raise awareness of this problem, I am not well versed with class loading, so I can’t really help there.
Sincerely,
Gunnar Brand
Gunnar Brand | Softwareentwickler
interface projects GmbH | www.intergator.de
Zwinglistraße 11/13 | 01277 Dresden | Deutschland
Tel: +49-351-31809-41 | Fax: +49-351-31809-33
Folgen Sie intergator auf:
Twitter : https://twitter.com/intergator
xing : https://www.xing.com/companies/interfaceprojectsgmbh
LinkedIn: http://www.linkedin.com/company/interface-projects-gmbh
YouTube : http://www.youtube.com/user/TheEnterpriseSearch?feature=watch
Facebook: https://www.facebook.com/intergator/
RSS : https://www.intergator.de/feed/
Geschäftsführer: Dr. Uwe Crenze, Eduard Daoud
Amtsgericht Dresden HRB 8740
Haftungsausschluss / Disclaimer
http://www.intergator.de/email-disclaimer.shtml
Re: Performance bottleneck in WebappClassLoaderBase
Posted by Mark Thomas <ma...@apache.org>.
This is a question, to start with at least, for the users mailing list.
Mark
On 25/09/2019 11:46, Gunnar Brand wrote:
> Hello.
>
>
>
> Recently we received complaints from a customer about slow response
> times (45 seconds) every morning. Every morning starting a fresh session
> in the browser would require a long wait time for the first user, but
> once this hurdle had been taken everything seemed to work fine.
>
> We could reproduce this in house, and doing a thread dump during this
> time span showed the class loader trying to open a jar file. Immediately
> we suspected a virus scanner slowing access to jar files (and since
> signatures are updated daily, this happened daily). Blocking the virus
> scanner from the tomcat folder helped but this is just alleviating the
> symptoms, not the cure. Doing a full trace log of the class loader
> package showed the problem:
>
> For any unknown resource or class the class loader seems to populate all
> known jars, and either finds it, then loads it, and keeps a cache entry
> (with the class for classes), or if it can’t find it, asks the parent
> class loader (this is the spec order of things).
>
> The log was filled classes and resources that couldn’t be found and were
> delegated to the parent. Over and over again. Each and every one of
> theses requests scans all jar files. Clear the OS cache and the problem
> appears, too.
>
>
>
> For classes one would believe this cannot be, once a class is loaded,
> it’s loaded. But with dynamic script languages, they often use
> Class.forName() and for example Rhino even does that for packages (that
> might not even exist!).
>
> We fixed at least Rhino with an application wide global top level , so
> for classes this problem did go away (there might be other, unless they
> implement their own caching, there is a risk).
>
> Resources are still a problem, our log is filled with
>
> 24-Sep-2019 17:22:55.482 FINE [http-nio-8080-exec-6]
> org.apache.catalina.loader.WebappClassLoaderBase.findResources
> findResources(META-INF/services/javax.xml.parsers.DocumentBuilderFactory)
>
> 24-Sep-2019 17:22:55.484 FINE [http-nio-8080-exec-6]
> org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream
> getResourceAsStream(META-INF/services/org.apache.xerces.xni.parser.XMLParserConfiguration)
>
> 24-Sep-2019 17:22:55.484 FINE [http-nio-8080-exec-6]
> org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream
> Searching local repositories
>
> 24-Sep-2019 17:22:55.485 FINE [http-nio-8080-exec-6]
> org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream
> Delegating to parent classloader unconditionally
> java.net.URLClassLoader@578486a3
>
> 24-Sep-2019 17:22:55.485 FINE [http-nio-8080-exec-6]
> org.apache.catalina.loader.WebappClassLoaderBase.getResourceAsStream
> --> Resource not found, returning null
>
> If there is only a single repeated “Searching local repositories” for a
> certain URL, the problem is there (doesn’t matter if the resource or
> class exists or not).
>
>
>
> It’s obviously a bad idea to scan the filesystem or jars every time.
>
> The WebappClassLoaderBase keeps information about every resource and
> class it finds, so repeated calls for these do not cost that much time,
> especially for classes that are stored in the cache.
>
> I believe it might work to keep a negative entry for classes it doesn’t
> find but the parent class loader knows and immediately ask the parent
> class loader. Maybe also for resources. But I don’t know what is
> supposed to happen if somebody overwrites a resource (or class) later.
>
> Even worse if the class/resource does not exist yet. Maybe populating
> the whole resource tree in memory is the only solution (with background
> update?!) .
>
>
>
> I mainly wanted to raise awareness of this problem, I am not well versed
> with class loading, so I can’t really help there.
>
>
>
> Sincerely,
>
> Gunnar Brand
>
>
>
>
>
>
>
> Gunnar Brand | Softwareentwickler
>
> interface projects GmbH | www.intergator.de
>
> Zwinglistraße 11/13 | 01277 Dresden | Deutschland
>
> Tel: +49-351-31809-41 | Fax: +49-351-31809-33
>
>
>
> Folgen Sie intergator auf:
>
> Twitter : https://twitter.com/intergator
>
> xing : https://www.xing.com/companies/interfaceprojectsgmbh
>
> LinkedIn: http://www.linkedin.com/company/interface-projects-gmbh
>
> YouTube : http://www.youtube.com/user/TheEnterpriseSearch?feature=watch
>
> Facebook: https://www.facebook.com/intergator/
>
> RSS : https://www.intergator.de/feed/
>
>
>
> Geschäftsführer: Dr. Uwe Crenze, Eduard Daoud
>
> Amtsgericht Dresden HRB 8740
>
>
>
> Haftungsausschluss / Disclaimer
>
> http://www.intergator.de/email-disclaimer.shtml
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org