You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Rhuberg,Anthony" <An...@Cerner.com.INVALID> on 2019/10/09 18:17:10 UTC
Performance test with Tomcat 9 shows increased cpu/disk usage because
of repeated opening/closing of jars in WEB-INF/lib
Background:
In the last few months we migrated our web application from Tomcat 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively straightforward until we reviewed the results of our performance tests. Those tests showed an increase in CPU usage and disk I/O on our Windows 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk usage were significantly lower (similar to previous tests).
As we investigated the cause of the increase in cpu/disk usage we found the tomcat process was repeatedly opening and closing the jars of our web application. The process profile showed all the jars in WEB-INF/lib being closed and reopened within seconds while the application was processing requests (and to some extent when quite). Now we are trying to determine why.
We have a general understanding that the WebappClassLoaderBase.java implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided for the configuration of jarOpenInterval (which we used the default = 90s). The latest implementation appears to track the jar modification times with a hash map called jarModificationTimes, but we are unable to find a similar time-to-live configuration or what triggers the lifecycle listener to close the jars (what calls WebappClassLoaderBase.modified() or destroy()) - assuming we understand this lifecycle behavior.
The server is configured mostly with default settings. The Context reloadable property is NOT set. We have tried increasing the cache time of the Resources cacheTtl property as the default of 5s seemed close to the time the jars were reloaded. We have not been successful.
Question:
What is the cause of the repeated reloading of our web application jars? What can trigger the class loader to close all the file handles related to the jars in the web application WEB-INF/lib directory - and then open them again within a few seconds?
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
I have not seen the trace message for modified()... So my understanding of this is wrong.
-----Original Message-----
From: Paul Carter-Brown <pa...@jini.guru>
Sent: Wednesday, October 09, 2019 4:42 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
Hi Anthony
Have you turned debug logging on to see what it is picking up as modified?
On Wed, 09 Oct 2019, 22:24 Rhuberg,Anthony, <An...@cerner.com.invalid> wrote:
> Thanks for your responses.
>
> I understand that re-reading the static files is not optimal, but how
> does reading files from the jar affect them being reloaded... In
> Tomcat 7, that seemed to prevent the jars from being reloaded since
> the accessTime was changed; in Tomcat 8, it seems the modified()
> method uses the file modified time - not sure what we are doing to
> cause the modified time to change - the files on the filesystem are
> not changing - unless there is something happening in the classes folder.
>
> Thanks
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Wednesday, October 09, 2019 3:56 PM
> To: users@tomcat.apache.org
> Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk
> usage because of repeated opening/closing of jars in WEB-INF/lib
>
> On 09/10/2019 20:08, Rhuberg,Anthony wrote:
> > On the other thread: Is this genuine class loading (in which case
> > the response below is correct) or is the application reading the
> > .class
> files as if they were resources (in which case a different answer applies)?
> > Mostly, we are talking about the default class loader; however, the
> application does read resources from the jar files - mostly static
> files that are stored within the package structure are read (for
> example: log4j, property files, other static content). Would this
> affect the reload of the jar files?
>
> If an application was reading the same resource from a JAR over and
> over again, that may have a performance impact. Increasing the TTL for
> cached resources would probably address that although, arguably, the
> better solution in that case would be to fix the app to load it once and cache it.
>
> > -----Original Message-----
> > From: Rhuberg,Anthony
> > Sent: Wednesday, October 09, 2019 2:53 PM
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk
> > usage because of repeated opening/closing of jars in WEB-INF/lib
> >
> > Just noticed another thread on this topic: RE: Tomcat discards and
> reloads the jar files from the webapps folder.
> >
> > Setting the Engine attribute backgroundProcessorDelay to 90s does
> > reduce
> the open/close cycle of the jars.
>
> Thanks for the confirmation.
>
> > The other thread also mentioned a potential configuration to keep
> > the
> jars open longer by default. This could be helpful - not sure we need
> to close them at all.
>
> If the JARs aren't closed it can cause file locking issues. We could
> include a disable option if we did make the Jar unload frequency
> configurable.
>
> > Alternatively, would there be any gain to 'explode' the jars into
> > class
> files on the filesystem?
>
> Maybe. Best bet it to try it for your app and see.
>
> Mark
>
>
> > -----Original Message-----
> > From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
> > Sent: Wednesday, October 09, 2019 2:17 PM
> > To: users@tomcat.apache.org
> > Subject: Performance test with Tomcat 9 shows increased cpu/disk
> > usage because of repeated opening/closing of jars in WEB-INF/lib
> >
> > Background:
> > In the last few months we migrated our web application from Tomcat
> 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively
> straightforward until we reviewed the results of our performance tests.
> Those tests showed an increase in CPU usage and disk I/O on our
> Windows
> 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk
> usage were significantly lower (similar to previous tests).
> >
> > As we investigated the cause of the increase in cpu/disk usage we
> > found
> the tomcat process was repeatedly opening and closing the jars of our
> web application. The process profile showed all the jars in
> WEB-INF/lib being closed and reopened within seconds while the
> application was processing requests (and to some extent when quite).
> Now we are trying to determine why.
> >
> > We have a general understanding that the WebappClassLoaderBase.java
> implementation changed in Tomcat 8.5. The Tomcat 7 implementation
> provided for the configuration of jarOpenInterval (which we used the
> default = 90s). The latest implementation appears to track the jar
> modification times with a hash map called jarModificationTimes, but we
> are unable to find a similar time-to-live configuration or what
> triggers the lifecycle listener to close the jars (what calls
> WebappClassLoaderBase.modified() or
> destroy()) - assuming we understand this lifecycle behavior.
> >
> > The server is configured mostly with default settings. The Context
> reloadable property is NOT set. We have tried increasing the cache
> time of the Resources cacheTtl property as the default of 5s seemed
> close to the time the jars were reloaded. We have not been successful.
> >
> > Question:
> > What is the cause of the repeated reloading of our web application jars?
> What can trigger the class loader to close all the file handles
> related to the jars in the web application WEB-INF/lib directory - and
> then open them again within a few seconds?
> >
> >
> >
> > CONFIDENTIALITY NOTICE This message and any included attachments are
> from Cerner Corporation and are intended only for the addressee. The
> information contained in this message is confidential and may
> constitute inside or non-public information under international,
> federal, or state securities laws. Unauthorized forwarding, printing,
> copying, distribution, or use of such information is strictly
> prohibited and may be unlawful. If you are not the addressee, please
> promptly delete this message and notify the sender of the delivery
> error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
Re: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by Paul Carter-Brown <pa...@jini.guru>.
Hi Anthony
Have you turned debug logging on to see what it is picking up as modified?
On Wed, 09 Oct 2019, 22:24 Rhuberg,Anthony,
<An...@cerner.com.invalid> wrote:
> Thanks for your responses.
>
> I understand that re-reading the static files is not optimal, but how does
> reading files from the jar affect them being reloaded... In Tomcat 7, that
> seemed to prevent the jars from being reloaded since the accessTime was
> changed; in Tomcat 8, it seems the modified() method uses the file modified
> time - not sure what we are doing to cause the modified time to change -
> the files on the filesystem are not changing - unless there is something
> happening in the classes folder.
>
> Thanks
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Wednesday, October 09, 2019 3:56 PM
> To: users@tomcat.apache.org
> Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage
> because of repeated opening/closing of jars in WEB-INF/lib
>
> On 09/10/2019 20:08, Rhuberg,Anthony wrote:
> > On the other thread: Is this genuine class loading (in which case the
> > response below is correct) or is the application reading the .class
> files as if they were resources (in which case a different answer applies)?
> > Mostly, we are talking about the default class loader; however, the
> application does read resources from the jar files - mostly static files
> that are stored within the package structure are read (for example: log4j,
> property files, other static content). Would this affect the reload of the
> jar files?
>
> If an application was reading the same resource from a JAR over and over
> again, that may have a performance impact. Increasing the TTL for cached
> resources would probably address that although, arguably, the better
> solution in that case would be to fix the app to load it once and cache it.
>
> > -----Original Message-----
> > From: Rhuberg,Anthony
> > Sent: Wednesday, October 09, 2019 2:53 PM
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk
> > usage because of repeated opening/closing of jars in WEB-INF/lib
> >
> > Just noticed another thread on this topic: RE: Tomcat discards and
> reloads the jar files from the webapps folder.
> >
> > Setting the Engine attribute backgroundProcessorDelay to 90s does reduce
> the open/close cycle of the jars.
>
> Thanks for the confirmation.
>
> > The other thread also mentioned a potential configuration to keep the
> jars open longer by default. This could be helpful - not sure we need to
> close them at all.
>
> If the JARs aren't closed it can cause file locking issues. We could
> include a disable option if we did make the Jar unload frequency
> configurable.
>
> > Alternatively, would there be any gain to 'explode' the jars into class
> files on the filesystem?
>
> Maybe. Best bet it to try it for your app and see.
>
> Mark
>
>
> > -----Original Message-----
> > From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
> > Sent: Wednesday, October 09, 2019 2:17 PM
> > To: users@tomcat.apache.org
> > Subject: Performance test with Tomcat 9 shows increased cpu/disk usage
> > because of repeated opening/closing of jars in WEB-INF/lib
> >
> > Background:
> > In the last few months we migrated our web application from Tomcat
> 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively
> straightforward until we reviewed the results of our performance tests.
> Those tests showed an increase in CPU usage and disk I/O on our Windows
> 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk
> usage were significantly lower (similar to previous tests).
> >
> > As we investigated the cause of the increase in cpu/disk usage we found
> the tomcat process was repeatedly opening and closing the jars of our web
> application. The process profile showed all the jars in WEB-INF/lib being
> closed and reopened within seconds while the application was processing
> requests (and to some extent when quite). Now we are trying to determine
> why.
> >
> > We have a general understanding that the WebappClassLoaderBase.java
> implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided
> for the configuration of jarOpenInterval (which we used the default =
> 90s). The latest implementation appears to track the jar modification times
> with a hash map called jarModificationTimes, but we are unable to find a
> similar time-to-live configuration or what triggers the lifecycle listener
> to close the jars (what calls WebappClassLoaderBase.modified() or
> destroy()) - assuming we understand this lifecycle behavior.
> >
> > The server is configured mostly with default settings. The Context
> reloadable property is NOT set. We have tried increasing the cache time of
> the Resources cacheTtl property as the default of 5s seemed close to the
> time the jars were reloaded. We have not been successful.
> >
> > Question:
> > What is the cause of the repeated reloading of our web application jars?
> What can trigger the class loader to close all the file handles related to
> the jars in the web application WEB-INF/lib directory - and then open them
> again within a few seconds?
> >
> >
> >
> > CONFIDENTIALITY NOTICE This message and any included attachments are
> from Cerner Corporation and are intended only for the addressee. The
> information contained in this message is confidential and may constitute
> inside or non-public information under international, federal, or state
> securities laws. Unauthorized forwarding, printing, copying, distribution,
> or use of such information is strictly prohibited and may be unlawful. If
> you are not the addressee, please promptly delete this message and notify
> the sender of the delivery error by e-mail or you may call Cerner's
> corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
Thanks for your responses.
I understand that re-reading the static files is not optimal, but how does reading files from the jar affect them being reloaded... In Tomcat 7, that seemed to prevent the jars from being reloaded since the accessTime was changed; in Tomcat 8, it seems the modified() method uses the file modified time - not sure what we are doing to cause the modified time to change - the files on the filesystem are not changing - unless there is something happening in the classes folder.
Thanks
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Wednesday, October 09, 2019 3:56 PM
To: users@tomcat.apache.org
Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
On 09/10/2019 20:08, Rhuberg,Anthony wrote:
> On the other thread: Is this genuine class loading (in which case the
> response below is correct) or is the application reading the .class files as if they were resources (in which case a different answer applies)?
> Mostly, we are talking about the default class loader; however, the application does read resources from the jar files - mostly static files that are stored within the package structure are read (for example: log4j, property files, other static content). Would this affect the reload of the jar files?
If an application was reading the same resource from a JAR over and over again, that may have a performance impact. Increasing the TTL for cached resources would probably address that although, arguably, the better solution in that case would be to fix the app to load it once and cache it.
> -----Original Message-----
> From: Rhuberg,Anthony
> Sent: Wednesday, October 09, 2019 2:53 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk
> usage because of repeated opening/closing of jars in WEB-INF/lib
>
> Just noticed another thread on this topic: RE: Tomcat discards and reloads the jar files from the webapps folder.
>
> Setting the Engine attribute backgroundProcessorDelay to 90s does reduce the open/close cycle of the jars.
Thanks for the confirmation.
> The other thread also mentioned a potential configuration to keep the jars open longer by default. This could be helpful - not sure we need to close them at all.
If the JARs aren't closed it can cause file locking issues. We could include a disable option if we did make the Jar unload frequency configurable.
> Alternatively, would there be any gain to 'explode' the jars into class files on the filesystem?
Maybe. Best bet it to try it for your app and see.
Mark
> -----Original Message-----
> From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
> Sent: Wednesday, October 09, 2019 2:17 PM
> To: users@tomcat.apache.org
> Subject: Performance test with Tomcat 9 shows increased cpu/disk usage
> because of repeated opening/closing of jars in WEB-INF/lib
>
> Background:
> In the last few months we migrated our web application from Tomcat 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively straightforward until we reviewed the results of our performance tests. Those tests showed an increase in CPU usage and disk I/O on our Windows 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk usage were significantly lower (similar to previous tests).
>
> As we investigated the cause of the increase in cpu/disk usage we found the tomcat process was repeatedly opening and closing the jars of our web application. The process profile showed all the jars in WEB-INF/lib being closed and reopened within seconds while the application was processing requests (and to some extent when quite). Now we are trying to determine why.
>
> We have a general understanding that the WebappClassLoaderBase.java implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided for the configuration of jarOpenInterval (which we used the default = 90s). The latest implementation appears to track the jar modification times with a hash map called jarModificationTimes, but we are unable to find a similar time-to-live configuration or what triggers the lifecycle listener to close the jars (what calls WebappClassLoaderBase.modified() or destroy()) - assuming we understand this lifecycle behavior.
>
> The server is configured mostly with default settings. The Context reloadable property is NOT set. We have tried increasing the cache time of the Resources cacheTtl property as the default of 5s seemed close to the time the jars were reloaded. We have not been successful.
>
> Question:
> What is the cause of the repeated reloading of our web application jars? What can trigger the class loader to close all the file handles related to the jars in the web application WEB-INF/lib directory - and then open them again within a few seconds?
>
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by Mark Thomas <ma...@apache.org>.
On 09/10/2019 20:08, Rhuberg,Anthony wrote:
> On the other thread: Is this genuine class loading (in which case the response below is correct) or is the application reading the .class files as if they were
> resources (in which case a different answer applies)?
> Mostly, we are talking about the default class loader; however, the application does read resources from the jar files - mostly static files that are stored within the package structure are read (for example: log4j, property files, other static content). Would this affect the reload of the jar files?
If an application was reading the same resource from a JAR over and over
again, that may have a performance impact. Increasing the TTL for cached
resources would probably address that although, arguably, the better
solution in that case would be to fix the app to load it once and cache it.
> -----Original Message-----
> From: Rhuberg,Anthony
> Sent: Wednesday, October 09, 2019 2:53 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
>
> Just noticed another thread on this topic: RE: Tomcat discards and reloads the jar files from the webapps folder.
>
> Setting the Engine attribute backgroundProcessorDelay to 90s does reduce the open/close cycle of the jars.
Thanks for the confirmation.
> The other thread also mentioned a potential configuration to keep the jars open longer by default. This could be helpful - not sure we need to close them at all.
If the JARs aren't closed it can cause file locking issues. We could
include a disable option if we did make the Jar unload frequency
configurable.
> Alternatively, would there be any gain to 'explode' the jars into class files on the filesystem?
Maybe. Best bet it to try it for your app and see.
Mark
> -----Original Message-----
> From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
> Sent: Wednesday, October 09, 2019 2:17 PM
> To: users@tomcat.apache.org
> Subject: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
>
> Background:
> In the last few months we migrated our web application from Tomcat 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively straightforward until we reviewed the results of our performance tests. Those tests showed an increase in CPU usage and disk I/O on our Windows 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk usage were significantly lower (similar to previous tests).
>
> As we investigated the cause of the increase in cpu/disk usage we found the tomcat process was repeatedly opening and closing the jars of our web application. The process profile showed all the jars in WEB-INF/lib being closed and reopened within seconds while the application was processing requests (and to some extent when quite). Now we are trying to determine why.
>
> We have a general understanding that the WebappClassLoaderBase.java implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided for the configuration of jarOpenInterval (which we used the default = 90s). The latest implementation appears to track the jar modification times with a hash map called jarModificationTimes, but we are unable to find a similar time-to-live configuration or what triggers the lifecycle listener to close the jars (what calls WebappClassLoaderBase.modified() or destroy()) - assuming we understand this lifecycle behavior.
>
> The server is configured mostly with default settings. The Context reloadable property is NOT set. We have tried increasing the cache time of the Resources cacheTtl property as the default of 5s seemed close to the time the jars were reloaded. We have not been successful.
>
> Question:
> What is the cause of the repeated reloading of our web application jars? What can trigger the class loader to close all the file handles related to the jars in the web application WEB-INF/lib directory - and then open them again within a few seconds?
>
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
We continue to profile our application and have found instances of using TransformerFactory.newInstance().
The more concerning problem is third party dependencies that do this or something similar we could not fix.
Jar modified times are not changing - I did not mean to state or otherwise imply that.
-----Original Message-----
From: John.E.Gregg@wellsfargo.com.INVALID <Jo...@wellsfargo.com.INVALID>
Sent: Thursday, October 10, 2019 6:54 PM
To: users@tomcat.apache.org
Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
Tony,
> -----Original Message-----
> From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
> Sent: Thursday, October 10, 2019 5:22 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk
> usage because of repeated opening/closing of jars in WEB-INF/lib
>
> We are still investigating what specific classloader reads that would
> trigger the repeated reload of the web-inf/lib/*.jars.
>
> One example we found is the use of
> javax.xml.transform.TransformerFactory.newInstance(). One of its
> features is to determine the implementation by searching for the
> configuration specified by
> META-INF/services/javax.xml.transform.TransformerFactory in jars available to the runtime.
>
Yep. I was wondering about that. I have no idea why this behavior would be different in Tomcat 9, though. Did you change Java versions also? Is the classpath longer?
I also have no idea about the jar modification date changes that you're seeing.
I do have first-hand experience with the performance of TransformerFactory.newInstance(). In general, anything that uses the service locator under the covers, such as TransformerFactory, DocumentBuilderFactory, etc, should be reused if possible. Those XML factories are thread safe, though the things created by the factories are typically not thread safe. You should create it once, configure it once, and use it as many times as necessary. For example, you would create one TransformerFactory, then call newTransformer() as many times as necessary.
John
B KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKCB [ X ܚX KK[XZ[
\ \ ][ X ܚX P X ]
\X K ܙ B ܈Y][ۘ[ [X[ K[XZ[
\ \ Z[ X ]
\X K ܙ B
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by Jo...@wellsfargo.com.INVALID.
Tony,
> -----Original Message-----
> From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
> Sent: Thursday, October 10, 2019 5:22 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage
> because of repeated opening/closing of jars in WEB-INF/lib
>
> We are still investigating what specific classloader reads that would trigger
> the repeated reload of the web-inf/lib/*.jars.
>
> One example we found is the use of
> javax.xml.transform.TransformerFactory.newInstance(). One of its features
> is to determine the implementation by searching for the configuration
> specified by META-INF/services/javax.xml.transform.TransformerFactory in
> jars available to the runtime.
>
Yep. I was wondering about that. I have no idea why this behavior would be different in Tomcat 9, though. Did you change Java versions also? Is the classpath longer?
I also have no idea about the jar modification date changes that you're seeing.
I do have first-hand experience with the performance of TransformerFactory.newInstance(). In general, anything that uses the service locator under the covers, such as TransformerFactory, DocumentBuilderFactory, etc, should be reused if possible. Those XML factories are thread safe, though the things created by the factories are typically not thread safe. You should create it once, configure it once, and use it as many times as necessary. For example, you would create one TransformerFactory, then call newTransformer() as many times as necessary.
John
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
We are still investigating what specific classloader reads that would trigger the repeated reload of the web-inf/lib/*.jars.
One example we found is the use of javax.xml.transform.TransformerFactory.newInstance(). One of its features is to determine the implementation by searching for the configuration specified by META-INF/services/javax.xml.transform.TransformerFactory in jars available to the runtime.
It seems users of Tomcat 9 would need to understand the management of the jar resource handles when using the TransformerFactory class - assuming it is triggering the reload of all the related application jars - which if happening enough can cause the high cpu/disk usage as we observed in our performance tests.
Thanks
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Thursday, October 10, 2019 3:54 AM
To: users@tomcat.apache.org
Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
On 09/10/2019 22:58, Rhuberg,Anthony wrote:
> StandardRoot.gc() unconditionally closes the web application jars in Tomcat 9... every 10s by default or configurable by changing backgroundProcessorDelay.
>
> So then the problem is.... Our web application is calling class.getResourceAsStream() to read some static files... and this is causing the jars to be read again?
It sounds like it.
> This was not a problem with Tomcat 7 because the jars were not closed if they were read within the jarOpenInterval setting?
Or that the frequency that the JARs were closed was low enough that the performance impact was not noticed.
Mark
>
> Thanks,
> Tony
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Wednesday, October 09, 2019 5:32 PM
> To: users@tomcat.apache.org
> Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk
> usage because of repeated opening/closing of jars in WEB-INF/lib
>
> On 09/10/2019 21:03, Rhuberg,Anthony wrote:
>> This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
>> <Context docBase="${STP_HOME}/app/webapps/Clinicals"
>> swallowOutput="true" backgroundProcessorDelay="90">
>>
>> Not sure if this is the context reload trigger... i.e. the webappLoader.backgroundProcess method is triggered every 90 seconds...
>
> It isn't. It is StandardRoot.gc() that is unloading the cached JAR entries.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
Hi,
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Thursday, October 10, 2019 3:54 AM
To: users@tomcat.apache.org
Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
On 09/10/2019 22:58, Rhuberg,Anthony wrote:
> StandardRoot.gc() unconditionally closes the web application jars in Tomcat 9... every 10s by default or configurable by changing backgroundProcessorDelay.
>
> So then the problem is.... Our web application is calling class.getResourceAsStream() to read some static files... and this is causing the jars to be read again?
It sounds like it.
> This was not a problem with Tomcat 7 because the jars were not closed if they were read within the jarOpenInterval setting?
Or that the frequency that the JARs were closed was low enough that the performance impact was not noticed.
Mark
>
> Thanks,
> Tony
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Wednesday, October 09, 2019 5:32 PM
> To: users@tomcat.apache.org
> Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk
> usage because of repeated opening/closing of jars in WEB-INF/lib
>
> On 09/10/2019 21:03, Rhuberg,Anthony wrote:
>> This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
>> <Context docBase="${STP_HOME}/app/webapps/Clinicals"
>> swallowOutput="true" backgroundProcessorDelay="90">
>>
>> Not sure if this is the context reload trigger... i.e. the webappLoader.backgroundProcess method is triggered every 90 seconds...
>
> It isn't. It is StandardRoot.gc() that is unloading the cached JAR entries.
>
> Mark
>
> ---------------------------------------------------------------------
Here is an update as to the causes of the repeated reloads our web application jars. Some of the following may be "bad" practices (at least not optimal), but seem to be commonplace. In other cases, the behavior was unexpected and may warrant a change to WebResourceRoot to allow the jars to remain open or otherwise control that behavior.
Have 2 questions:
1. Could the WebResourceRoot be changed to provide a way to control when the jars are closed?
2. Any concerns with any of the solutions below? Has anyone used these in the past or currently and found any problems?
1. JAXP related factories
We frequently use DocumentBuilderFactory.newInstance() to obtain an instance of the factory implementation and others like TransformerFactory, XPathFactory, DatatypeFactory, and SAXParserFactory. This will trigger a jar reload when the service loader attempts to find the implementation class. We decided to set the applicable system properties to avoid the search and thus eliminate this reason for the jar reload. See below for the factories configured; this change reduced JVM CPU usage percentage in a load test by 26% (using Tomcat 7, so expect similar in Tomcat 9, plus the elimination of the jar reload).
2. SOAP web service endpoints
Each time we invoke a SOAP web service we are creating an instance of javax.xml.ws.Service and a client proxy that implements the applicable web service contract (Service.getPort). When these objects are created the service loader is used to find the implementation for the web service client. We are using the Apache CXF client (cxf-rt-frontend-jaxws/META-INF/services/javax.xml.ws.spi.Provider). In addition, the JAXB bindings are created. These actions occur every time we invoke a SOAP web service; this subsequently searches for the implementation classes triggering the web application jars to be reloaded. We decided to create the service end-points once and reuse. Creating the objects once and reusing avoids the cost of repeatedly creating the instances. The CXF instances are thread safe (https://cxf.apache.org/faq.html#FAQ-AreJAX-WSclientproxiesthreadsafe?). Testing to verify the objects are thread safe is in progress.
3. When we invoke a REST web service, we create an instance of the Spring RestTemplate. The template then registers various mappers; the MappingJackson2HttpMessageConverter uses Jackson2ObjectMapperBuilder.registerWellKnownModulesIfAvailable() to register a few optional modules. Each time the template is created a classpath search is performed that can trigger the jars to be reloaded. See: https://github.com/spring-projects/spring-framework/search?utf8=%E2%9C%93&q=registerWellKnownModulesIfAvailable&type=. The optional modules include: Jdk7Module, Jdk8Module, and JavaModule. Adding these jars allows the classes to be found and then cached, thus preventing subsequent jar reloads: jackson-datatype-jsr310-2.10.1.jar
jackson-datatype-jdk7-2.6.7.jar, and jackson-datatype-jdk8-2.10.1.jar.
4. When transforming XML documents, we repeatedly read the .xsl file using getResourceAsStream(sometransform.xsl). This is cached by default for 5 seconds. We increased the associated Resources cacheTtl to 60 minutes. Not sure if this will be our solution.
5. We implemented an override to StandardRoot to simply no-op the gc() operation; then we configure the related Resources to use that class. We have not decided to use this; it helps set the performance baseline back to the behavior we saw in Tomcat 7.
Related to #1 above...
JAXP properties
javax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl
javax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl
javax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl
javax.xml.datatype.DatatypeFactory=com.sun.org.apache.xerces.internal.jaxp.datatype.DatatypeFactoryImpl
javax.xml.stream.XMLInputFactory=com.sun.xml.internal.stream.XMLInputFactoryImpl
javax.xml.stream.XMLOutputFactory=com.sun.xml.internal.stream.XMLOutputFactoryImpl
javax.xml.stream.XMLEventFactory=com.sun.xml.internal.stream.events.XMLEventFactoryImpl
javax.xml.xpath.XPathFactory\:http\://java.sun.com/jaxp/xpath/dom=com.sun.org.apache.xpath.internal.jaxp.XPathFactoryImpl
javax.xml.validation.SchemaFactory\:http\://www.w3.org/2001/XMLSchema=com.sun.org.apache.xerces.internal.jaxp.validation.XMLSchemaFactory
org.apache.xml.dtm.DTMManager=org.apache.xml.dtm.ref.DTMManagerDefault
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
Re: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by Mark Thomas <ma...@apache.org>.
On 09/10/2019 22:58, Rhuberg,Anthony wrote:
> StandardRoot.gc() unconditionally closes the web application jars in Tomcat 9... every 10s by default or configurable by changing backgroundProcessorDelay.
>
> So then the problem is.... Our web application is calling class.getResourceAsStream() to read some static files... and this is causing the jars to be read again?
It sounds like it.
> This was not a problem with Tomcat 7 because the jars were not closed if they were read within the jarOpenInterval setting?
Or that the frequency that the JARs were closed was low enough that the
performance impact was not noticed.
Mark
>
> Thanks,
> Tony
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Wednesday, October 09, 2019 5:32 PM
> To: users@tomcat.apache.org
> Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
>
> On 09/10/2019 21:03, Rhuberg,Anthony wrote:
>> This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
>> <Context docBase="${STP_HOME}/app/webapps/Clinicals"
>> swallowOutput="true" backgroundProcessorDelay="90">
>>
>> Not sure if this is the context reload trigger... i.e. the webappLoader.backgroundProcess method is triggered every 90 seconds...
>
> It isn't. It is StandardRoot.gc() that is unloading the cached JAR entries.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
StandardRoot.gc() unconditionally closes the web application jars in Tomcat 9... every 10s by default or configurable by changing backgroundProcessorDelay.
So then the problem is.... Our web application is calling class.getResourceAsStream() to read some static files... and this is causing the jars to be read again? This was not a problem with Tomcat 7 because the jars were not closed if they were read within the jarOpenInterval setting?
Thanks,
Tony
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Wednesday, October 09, 2019 5:32 PM
To: users@tomcat.apache.org
Subject: Re: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
On 09/10/2019 21:03, Rhuberg,Anthony wrote:
> This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
> <Context docBase="${STP_HOME}/app/webapps/Clinicals"
> swallowOutput="true" backgroundProcessorDelay="90">
>
> Not sure if this is the context reload trigger... i.e. the webappLoader.backgroundProcess method is triggered every 90 seconds...
It isn't. It is StandardRoot.gc() that is unloading the cached JAR entries.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
Re: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by Mark Thomas <ma...@apache.org>.
On 09/10/2019 21:03, Rhuberg,Anthony wrote:
> This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
> <Context docBase="${STP_HOME}/app/webapps/Clinicals" swallowOutput="true" backgroundProcessorDelay="90">
>
> Not sure if this is the context reload trigger... i.e. the webappLoader.backgroundProcess method is triggered every 90 seconds...
It isn't. It is StandardRoot.gc() that is unloading the cached JAR entries.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
This seems to alleviate the issue... in context.xml (sc-test#sc.xml)
<Context docBase="${STP_HOME}/app/webapps/Clinicals" swallowOutput="true" backgroundProcessorDelay="90">
Not sure if this is the context reload trigger... i.e. the webappLoader.backgroundProcess method is triggered every 90 seconds... the reloadable property on Context is NOT set (default = false). So then how does modified() evaluate to true?
The https://github.com/apache/tomcat/blob/65aec2ac7fb4980694291e66efd360195dfa8d73/java/org/apache/catalina/loader/WebappLoader.java...
/**
* Execute a periodic task, such as reloading, etc. This method will be
* invoked inside the classloading context of this container. Unexpected
* throwables will be caught and logged.
*/
@Override
public void backgroundProcess() {
if (reloadable && modified()) {
try {
Thread.currentThread().setContextClassLoader
(WebappLoader.class.getClassLoader());
if (context != null) {
context.reload();
}
} finally {
if (context != null && context.getLoader() != null) {
Thread.currentThread().setContextClassLoader
(context.getLoader().getClassLoader());
}
}
}
}
I do not understand how modified = true. Why would the jar date/time be changed?
https://github.com/apache/tomcat/blob/8.5.x/java/org/apache/catalina/loader/WebappClassLoaderBase.java
/**
* Have one or more classes or resources been modified so that a reload
* is appropriate?
* @return <code>true</code> if there's been a modification
*/
public boolean modified() {
if (log.isDebugEnabled())
log.debug("modified()");
for (Entry<String,ResourceEntry> entry : resourceEntries.entrySet()) {
long cachedLastModified = entry.getValue().lastModified;
long lastModified = resources.getClassLoaderResource(
entry.getKey()).getLastModified();
if (lastModified != cachedLastModified) {
if( log.isDebugEnabled() )
log.debug(sm.getString("webappClassLoader.resourceModified",
entry.getKey(),
new Date(cachedLastModified),
new Date(lastModified)));
return true;
}
}
// Check if JARs have been added or removed
WebResource[] jars = resources.listResources("/WEB-INF/lib");
// Filter out non-JAR resources
int jarCount = 0;
for (WebResource jar : jars) {
if (jar.getName().endsWith(".jar") && jar.isFile() && jar.canRead()) {
jarCount++;
Long recordedLastModified = jarModificationTimes.get(jar.getName());
if (recordedLastModified == null) {
// Jar has been added
log.info(sm.getString("webappClassLoader.jarsAdded",
resources.getContext().getName()));
return true;
}
if (recordedLastModified.longValue() != jar.getLastModified()) {
// Jar has been changed
log.info(sm.getString("webappClassLoader.jarsModified",
resources.getContext().getName()));
return true;
}
}
}
if (jarCount < jarModificationTimes.size()){
log.info(sm.getString("webappClassLoader.jarsRemoved",
resources.getContext().getName()));
return true;
}
// No classes have been modified
return false;
}
-----Original Message-----
From: Rhuberg,Anthony
Sent: Wednesday, October 09, 2019 3:08 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
On the other thread: Is this genuine class loading (in which case the response below is correct) or is the application reading the .class files as if they were resources (in which case a different answer applies)?
Mostly, we are talking about the default class loader; however, the application does read resources from the jar files - mostly static files that are stored within the package structure are read (for example: log4j, property files, other static content). Would this affect the reload of the jar files?
-----Original Message-----
From: Rhuberg,Anthony
Sent: Wednesday, October 09, 2019 2:53 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
Just noticed another thread on this topic: RE: Tomcat discards and reloads the jar files from the webapps folder.
Setting the Engine attribute backgroundProcessorDelay to 90s does reduce the open/close cycle of the jars.
The other thread also mentioned a potential configuration to keep the jars open longer by default. This could be helpful - not sure we need to close them at all.
Alternatively, would there be any gain to 'explode' the jars into class files on the filesystem?
-----Original Message-----
From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
Sent: Wednesday, October 09, 2019 2:17 PM
To: users@tomcat.apache.org
Subject: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
Background:
In the last few months we migrated our web application from Tomcat 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively straightforward until we reviewed the results of our performance tests. Those tests showed an increase in CPU usage and disk I/O on our Windows 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk usage were significantly lower (similar to previous tests).
As we investigated the cause of the increase in cpu/disk usage we found the tomcat process was repeatedly opening and closing the jars of our web application. The process profile showed all the jars in WEB-INF/lib being closed and reopened within seconds while the application was processing requests (and to some extent when quite). Now we are trying to determine why.
We have a general understanding that the WebappClassLoaderBase.java implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided for the configuration of jarOpenInterval (which we used the default = 90s). The latest implementation appears to track the jar modification times with a hash map called jarModificationTimes, but we are unable to find a similar time-to-live configuration or what triggers the lifecycle listener to close the jars (what calls WebappClassLoaderBase.modified() or destroy()) - assuming we understand this lifecycle behavior.
The server is configured mostly with default settings. The Context reloadable property is NOT set. We have tried increasing the cache time of the Resources cacheTtl property as the default of 5s seemed close to the time the jars were reloaded. We have not been successful.
Question:
What is the cause of the repeated reloading of our web application jars? What can trigger the class loader to close all the file handles related to the jars in the web application WEB-INF/lib directory - and then open them again within a few seconds?
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
On the other thread: Is this genuine class loading (in which case the response below is correct) or is the application reading the .class files as if they were
resources (in which case a different answer applies)?
Mostly, we are talking about the default class loader; however, the application does read resources from the jar files - mostly static files that are stored within the package structure are read (for example: log4j, property files, other static content). Would this affect the reload of the jar files?
-----Original Message-----
From: Rhuberg,Anthony
Sent: Wednesday, October 09, 2019 2:53 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: RE: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
Just noticed another thread on this topic: RE: Tomcat discards and reloads the jar files from the webapps folder.
Setting the Engine attribute backgroundProcessorDelay to 90s does reduce the open/close cycle of the jars.
The other thread also mentioned a potential configuration to keep the jars open longer by default. This could be helpful - not sure we need to close them at all.
Alternatively, would there be any gain to 'explode' the jars into class files on the filesystem?
-----Original Message-----
From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
Sent: Wednesday, October 09, 2019 2:17 PM
To: users@tomcat.apache.org
Subject: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
Background:
In the last few months we migrated our web application from Tomcat 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively straightforward until we reviewed the results of our performance tests. Those tests showed an increase in CPU usage and disk I/O on our Windows 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk usage were significantly lower (similar to previous tests).
As we investigated the cause of the increase in cpu/disk usage we found the tomcat process was repeatedly opening and closing the jars of our web application. The process profile showed all the jars in WEB-INF/lib being closed and reopened within seconds while the application was processing requests (and to some extent when quite). Now we are trying to determine why.
We have a general understanding that the WebappClassLoaderBase.java implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided for the configuration of jarOpenInterval (which we used the default = 90s). The latest implementation appears to track the jar modification times with a hash map called jarModificationTimes, but we are unable to find a similar time-to-live configuration or what triggers the lifecycle listener to close the jars (what calls WebappClassLoaderBase.modified() or destroy()) - assuming we understand this lifecycle behavior.
The server is configured mostly with default settings. The Context reloadable property is NOT set. We have tried increasing the cache time of the Resources cacheTtl property as the default of 5s seemed close to the time the jars were reloaded. We have not been successful.
Question:
What is the cause of the repeated reloading of our web application jars? What can trigger the class loader to close all the file handles related to the jars in the web application WEB-INF/lib directory - and then open them again within a few seconds?
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Performance test with Tomcat 9 shows increased cpu/disk usage
because of repeated opening/closing of jars in WEB-INF/lib
Posted by "Rhuberg,Anthony" <An...@Cerner.com.INVALID>.
Just noticed another thread on this topic: RE: Tomcat discards and reloads the jar files from the webapps folder.
Setting the Engine attribute backgroundProcessorDelay to 90s does reduce the open/close cycle of the jars.
The other thread also mentioned a potential configuration to keep the jars open longer by default. This could be helpful - not sure we need to close them at all.
Alternatively, would there be any gain to 'explode' the jars into class files on the filesystem?
-----Original Message-----
From: Rhuberg,Anthony <An...@Cerner.com.INVALID>
Sent: Wednesday, October 09, 2019 2:17 PM
To: users@tomcat.apache.org
Subject: Performance test with Tomcat 9 shows increased cpu/disk usage because of repeated opening/closing of jars in WEB-INF/lib
Background:
In the last few months we migrated our web application from Tomcat 7.0.55 to Tomcat 9.0.19 (26). That transition was relatively straightforward until we reviewed the results of our performance tests. Those tests showed an increase in CPU usage and disk I/O on our Windows 2012 server. When we switch back to Tomcat 7, the metrics for CPU/disk usage were significantly lower (similar to previous tests).
As we investigated the cause of the increase in cpu/disk usage we found the tomcat process was repeatedly opening and closing the jars of our web application. The process profile showed all the jars in WEB-INF/lib being closed and reopened within seconds while the application was processing requests (and to some extent when quite). Now we are trying to determine why.
We have a general understanding that the WebappClassLoaderBase.java implementation changed in Tomcat 8.5. The Tomcat 7 implementation provided for the configuration of jarOpenInterval (which we used the default = 90s). The latest implementation appears to track the jar modification times with a hash map called jarModificationTimes, but we are unable to find a similar time-to-live configuration or what triggers the lifecycle listener to close the jars (what calls WebappClassLoaderBase.modified() or destroy()) - assuming we understand this lifecycle behavior.
The server is configured mostly with default settings. The Context reloadable property is NOT set. We have tried increasing the cache time of the Resources cacheTtl property as the default of 5s seemed close to the time the jars were reloaded. We have not been successful.
Question:
What is the cause of the repeated reloading of our web application jars? What can trigger the class loader to close all the file handles related to the jars in the web application WEB-INF/lib directory - and then open them again within a few seconds?
CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org