You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by William Crowell <WC...@perforce.com.INVALID> on 2023/11/06 17:55:00 UTC
Are there any known class loader leaks in Tomcat 9?
Good afternoon,
I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base.
Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check.
Regards,
William Crowell
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Are there any known class loader leaks in Tomcat 9?
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hello William,
On Mon, Nov 6, 2023 at 11:25 PM William Crowell
<WC...@perforce.com.invalid> wrote:
>
> Good afternoon,
>
> I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base.
>
How are you loading those large classes? using some custom classloader
OR Tomcat's normal classloader? Also, are you using a single
classloader to load all/multiple classes? Reason I asked is, until ALL
classes loaded by a classloader are de-referenced, the entire set of
classes loaded by the classloader will NOT be garbage collected. Most
possibly that is what is happening in your case.
Also, how fast Metaspace is growing? I will suggest checking the
contents of Metaspace by taking class stats details periodically.
GC.class_stats of Jcmd is helpful.
> Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check.
>
I have been using Tomcat 9 for the last two years and have not
experienced any problem where Tomcat is holding onto classes.
> Regards,
>
> William Crowell
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Are there any known class loader leaks in Tomcat 9?
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hello William,
On Tue, Nov 7, 2023 at 4:29 PM William Crowell
<WC...@perforce.com.invalid> wrote:
>
> Olaf and Sevendu,
>
> Thank you for your replies. Correct, I sincerely doubt this is a Tomcat class loading bug.
>
> I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into memory, and it is a single class loader.
>
> I am going to periodically run: jcmd <pid> GC.class_stats
>
> I am only having the issue in general operation and not on a redeploy, and I have to restart the server daily. I will find out more details as to how these classes are loaded into memory.
>
> Here is what is going on…
>
> I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at 8GB. JVM arguments are:
>
> -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
>
> The MaxMetaspaceSize is not set, so this means it is unlimited.
>
> After the server comes up and after a short period of time I get a native out of memory condition:
>
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
> # Possible reasons:
> # The system is out of physical RAM or swap space
> # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
> # Possible solutions:
> # Reduce memory load on the system
> # Increase physical memory or swap space
> # Check if swap backing store is full
> # Decrease Java heap size (-Xmx/-Xms)
> # Decrease number of Java threads
> # Decrease Java thread stack sizes (-Xss)
> # Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x000014fff94b7700
> #
> # JRE version: (8.0_371) (build )
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 compressed oops)
> # Core dump written. Default location: /hosting/core or core.191803
> #
>
So, it is not a Metaspace OOM. It can happen that Metaspace or some
other segment of JVM is increasing with time. Since you have NMT
enabled(-XX:NativeMemoryTracking=detail), please collect native memory
stats periodically until JVM crashes. After that, if you compare them,
you should see one of the segments is growing with time. Then you can
focus on that area. jcmd <pid> VM.native_memory can help with this.
> Regards,
>
> William Crowell
>
>
> From: Olaf Kock <to...@olafkock.de>
> Date: Tuesday, November 7, 2023 at 4:25 AM
> To: users@tomcat.apache.org <us...@tomcat.apache.org>
> Subject: Re: Are there any known class loader leaks in Tomcat 9?
>
> On 06.11.23 18:55, William Crowell wrote:
> > Good afternoon,
> >
> > I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base.
> >
> > Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check.
>
> Hi William,
>
> when you say "I am loading some very large Java classes into Metaspace.
> I know this is not good practice" it does not sound like this inherited
> library is using tomcat's standard way of classloading - please clarify.
>
> Also, nothing in that statement points to anything where a leak on
> tomcat's side would make a difference: Are you experiencing problems
> after reloading a webapp, or just in general operation? You might simply
> need to assign more metaspace to the VM for this library to operate
> properly.
>
> So, this boils down to:
>
> * How does this library load the classes into memory?
>
> * Do you redeploy?
>
> * What's your metaspace (and general memory) setting, and what are the
> conditions under which you run out of memory?
>
> * Do you see any log message that hint at classes that won't be garbage
> collected because of stale references? Tomcat issues those warnings, in
> case you (for example) start your own background threads that keep
> holding the memory allocated to them.
>
> Also: I second your doubt about Tomcat's classloader being the cause for
> your problems - unless proven otherwise.
>
> Olaf
>
>
>
>
> CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
>
> This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Are there any known class loader leaks in Tomcat 9?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
William,
On 11/7/23 05:59, William Crowell wrote:
> Olaf and Sevendu,
>
> Thank you for your replies. Correct, I sincerely doubt this is a Tomcat class loading bug.
>
> I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into memory, and it is a single class loader.
>
> I am going to periodically run: jcmd <pid> GC.class_stats
>
> I am only having the issue in general operation and not on a redeploy, and I have to restart the server daily. I will find out more details as to how these classes are loaded into memory.
>
> Here is what is going on…
>
> I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at 8GB. JVM arguments are:
>
> -XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
>
> The MaxMetaspaceSize is not set, so this means it is unlimited.
>
> After the server comes up and after a short period of time I get a native out of memory condition:
>
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
> # Possible reasons:
> # The system is out of physical RAM or swap space
> # The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
> # Possible solutions:
> # Reduce memory load on the system
> # Increase physical memory or swap space
> # Check if swap backing store is full
> # Decrease Java heap size (-Xmx/-Xms)
> # Decrease number of Java threads
> # Decrease Java thread stack sizes (-Xss)
> # Set larger code cache with -XX:ReservedCodeCacheSize=
> # This output file may be truncated or incomplete.
> #
> # Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x000014fff94b7700
> #
> # JRE version: (8.0_371) (build )
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 compressed oops)
> # Core dump written. Default location: /hosting/core or core.191803
> #
Enable heap dumps on OOME and look at them with a heap analyzer. You may
find that you have some huge things in memory that aren't being released
for other reasons.
If you have to bounce your server every day, I suspect that you have a
"known resource leak" in your web application already. It's very
unlikely to be Tomcat doing this.
What you should NOT do is just keep raising the heap size until it stops
failing, because you will never find out what is taking up all that heap
space.
If you find that nothing is out of place in that heap dump then AND ONLY
then should you raise the heap size. Sometimes you really just need more
heap.
-chri
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Are there any known class loader leaks in Tomcat 9?
Posted by William Crowell <WC...@perforce.com.INVALID>.
Olaf and Sevendu,
Thank you for your replies. Correct, I sincerely doubt this is a Tomcat class loading bug.
I am using Tomcat’s normal class loader (webapp/WAR) to load the classes into memory, and it is a single class loader.
I am going to periodically run: jcmd <pid> GC.class_stats
I am only having the issue in general operation and not on a redeploy, and I have to restart the server daily. I will find out more details as to how these classes are loaded into memory.
Here is what is going on…
I have a 16GB Linux instance and one Apache Tomcat 9.0.78 instance running on it. It is running JDK 1.8.0_371-b25. I have min and max JVM heap setting at 8GB. JVM arguments are:
-XX:CICompilerCount=3 -XX:ConcGCThreads=1 -XX:+DisableExplicitGC -XX:G1HeapRegionSize=4194304 -XX:GCLogFileSize=3145728 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=8589934592 -XX:MarkStackSize=4194304 -XX:MaxHeapSize=8589934592 -XX:MaxNewSize=5150605312 -XX:MinHeapDeltaBytes=4194304 -XX:NativeMemoryTracking=detail -XX:NumberOfGCLogFiles=25 -XX:+PrintGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseFastUnorderedTimeStamps -XX:+UseG1GC -XX:+UseGCLogFileRotation
The MaxMetaspaceSize is not set, so this means it is unlimited.
After the server comes up and after a short period of time I get a native out of memory condition:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2798), pid=191803, tid=0x000014fff94b7700
#
# JRE version: (8.0_371) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.371-b25 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /hosting/core or core.191803
#
Regards,
William Crowell
From: Olaf Kock <to...@olafkock.de>
Date: Tuesday, November 7, 2023 at 4:25 AM
To: users@tomcat.apache.org <us...@tomcat.apache.org>
Subject: Re: Are there any known class loader leaks in Tomcat 9?
On 06.11.23 18:55, William Crowell wrote:
> Good afternoon,
>
> I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base.
>
> Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check.
Hi William,
when you say "I am loading some very large Java classes into Metaspace.
I know this is not good practice" it does not sound like this inherited
library is using tomcat's standard way of classloading - please clarify.
Also, nothing in that statement points to anything where a leak on
tomcat's side would make a difference: Are you experiencing problems
after reloading a webapp, or just in general operation? You might simply
need to assign more metaspace to the VM for this library to operate
properly.
So, this boils down to:
* How does this library load the classes into memory?
* Do you redeploy?
* What's your metaspace (and general memory) setting, and what are the
conditions under which you run out of memory?
* Do you see any log message that hint at classes that won't be garbage
collected because of stale references? Tomcat issues those warnings, in
case you (for example) start your own background threads that keep
holding the memory allocated to them.
Also: I second your doubt about Tomcat's classloader being the cause for
your problems - unless proven otherwise.
Olaf
CAUTION: This email originated from outside of the organization. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.
Re: Are there any known class loader leaks in Tomcat 9?
Posted by Olaf Kock <to...@olafkock.de>.
On 06.11.23 18:55, William Crowell wrote:
> Good afternoon,
>
> I am running Tomcat 9.0.78 with JDK 1.8.0_371 (running with G1GC), and I am loading some very large Java classes into Metaspace. I know this is not good practice, but I inherited this library. These classes have business rules and are doing some basic primitive and array manipulations, and I am running out of native memory. I don’t think there are any JNI calls in this code base.
>
> Are there anything existing issues with the Tomcat 9 class loader? I doubt there are but wanted to check.
Hi William,
when you say "I am loading some very large Java classes into Metaspace.
I know this is not good practice" it does not sound like this inherited
library is using tomcat's standard way of classloading - please clarify.
Also, nothing in that statement points to anything where a leak on
tomcat's side would make a difference: Are you experiencing problems
after reloading a webapp, or just in general operation? You might simply
need to assign more metaspace to the VM for this library to operate
properly.
So, this boils down to:
* How does this library load the classes into memory?
* Do you redeploy?
* What's your metaspace (and general memory) setting, and what are the
conditions under which you run out of memory?
* Do you see any log message that hint at classes that won't be garbage
collected because of stale references? Tomcat issues those warnings, in
case you (for example) start your own background threads that keep
holding the memory allocated to them.
Also: I second your doubt about Tomcat's classloader being the cause for
your problems - unless proven otherwise.
Olaf