You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Mark A. Claassen" <MC...@ocie.net> on 2021/07/08 18:32:42 UTC
RE: HTTP/2 Memory Leak
Sorry, realized I had a mistyped subject.
-----Original Message-----
From: Mark A. Claassen <MC...@ocie.net>
Sent: Thursday, July 8, 2021 2:30 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Http/s Memory Leak
Importance: Low
We are using 9.0.12 on a server and noticed a pretty big memory leak. The change log mentions a some fixed leaks in the releases since 9.0.12.
I was wondering if this leak was fixed already.
The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes. Below is the reference path.
Is this one of the things fixed by setting useAsyncIO="false"? Is there a downside to using this?
(We are using Http11AprProtocol and OpenSSL)
Thanks,
Mark
Reference path of the leak:
MessageBytes
MimeHeaderField
MimeHeaderField[]
MimeHeaders
Response
Stream
ConcurrentHashMap$Node
ConcurrentHashMap$Node
ConcurrentHashMap$Node[]
Http2UpgradeHandler
StreamProcessor (Root)
Mark Claassen
Senior Software Engineer
Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN 46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014
Disclaimer:
The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thanks for your help, and all the help you provide to people on this list. Your replies are always quick, professional, and helpful.
A big part of what makes Tomcat so great is the assistance that can be found on this list!
Mark Claassen
Senior Software Engineer
Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN 46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014
Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Friday, July 9, 2021 1:31 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
On 09/07/2021 18:23, Mark A. Claassen wrote:
> Thank you so much! This is good to know.
>
> We may not change anything then (until that server gets upgraded). However, would switching back to HTTP 1.1 reduce the memory footprint in the mean time?
Yes.
The issue is the HTTP/2 priority tree. Essentially, Tomcat has to keep some information around about the previous few hundred requests on every
HTTP/2 connection. Until .37/.39 Tomcat just kept objects used for the request/response. As of .37/.39 Tomcat replaces these with a much lighter object once the request is completed.
> Would the connection be cleaned up when the browser goes away?
Should be.
Mark
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN 46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> -------------------------------------------
> Confidentiality Notice: OCIESERVICE
> -------------------------------------------
> The contents of this e-mail message and any attachments are intended solely for the addressee(s) named in this message. This communication is intended to be and to remain confidential. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not deliver, distribute, copy, disclose the contents or take any action in reliance upon the information contained in the communication or any attachments.
>
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Friday, July 9, 2021 12:59 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
>
> On 09/07/2021 16:21, Mark A. Claassen wrote:
>> Thanks. I have done more heap analysis and think I have it tracked closer to the source.
>>
>> --
>> I started looking at the heap a different way. The random values I looked at before (of the 80,000) may not have be as representative as I thought.
>>
>> Examining the retained sizes in the heap, I am finding:
>> I have two instances of AbstractProtocol$ConnectionHandler.
>>
>> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
>> This map as 32 elements in its "table". Most of these are null. Some of the ones that are not, are huge.
>> The entirety of the map retains 112MG.
>>
>> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
>> Looking at the "val" variable of a node, there is an
>> UpgradeProcessorInternal Inside this is a variable called
>> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I am looking at now retains 16MG of memory.
>> (Oddly, once I get this far, the retained sizes of its internal
>> objects don't really add up.)
>>
>> Any ideas on how to work around this? Or if this is already fixed in a later version of Tomcat?
>
> That isn't a leak. That is high memory usage. The objects will be GC'd once the HTTP/2 connection closes.
>
> The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.
>
> Mark
>
>
>>
>> Thanks,
>>
>> Mark Claassen
>> Senior Software Engineer
>>
>> Donnell Systems, Inc.
>> 130 South Main Street
>> Leighton Plaza Suite 375
>> South Bend, IN 46601
>> E-mail: mailto:mclaassen@ocie.net
>> Voice: (574)232-3784
>> Fax: (574)232-4014
>>
>> Disclaimer:
>> The opinions provided herein do not necessarily state or reflect
>> those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
>> assumes no legal liability or responsibility for the posting.
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rob Sargent <rs...@xmission.com>
>> Sent: Thursday, July 8, 2021 6:50 PM
>> To: users@tomcat.apache.org
>> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
>> Importance: Low
>>
>>
>>
>> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>>> Ok. That didn’t seem to work. I will investigate further and try to find a way to send that information.
>>>
>>> It is not that busy a server, but the memory use increases very quickly. Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>>
>>> (We have a temporary monitor script in place that does a GC and then
>>> prints a class_histogram every half hour to help us pinpoint what is
>>> happening.)
>>>
>>> Thanks,
>>> Mark
>>>
>> Perhaps you've done this already, but
>>
>> grep -R 'static HashMap' src
>>
>> might locate some potential culprits.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by Mark Thomas <ma...@apache.org>.
On 09/07/2021 18:23, Mark A. Claassen wrote:
> Thank you so much! This is good to know.
>
> We may not change anything then (until that server gets upgraded). However, would switching back to HTTP 1.1 reduce the memory footprint in the mean time?
Yes.
The issue is the HTTP/2 priority tree. Essentially, Tomcat has to keep
some information around about the previous few hundred requests on every
HTTP/2 connection. Until .37/.39 Tomcat just kept objects used for the
request/response. As of .37/.39 Tomcat replaces these with a much
lighter object once the request is completed.
> Would the connection be cleaned up when the browser goes away?
Should be.
Mark
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN 46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> -------------------------------------------
> Confidentiality Notice: OCIESERVICE
> -------------------------------------------
> The contents of this e-mail message and any attachments are intended solely for the addressee(s) named in this message. This communication is intended to be and to remain confidential. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not deliver, distribute, copy, disclose the contents or take any action in reliance upon the information contained in the communication or any attachments.
>
>
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Friday, July 9, 2021 12:59 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
>
> On 09/07/2021 16:21, Mark A. Claassen wrote:
>> Thanks. I have done more heap analysis and think I have it tracked closer to the source.
>>
>> --
>> I started looking at the heap a different way. The random values I looked at before (of the 80,000) may not have be as representative as I thought.
>>
>> Examining the retained sizes in the heap, I am finding:
>> I have two instances of AbstractProtocol$ConnectionHandler.
>>
>> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
>> This map as 32 elements in its "table". Most of these are null. Some of the ones that are not, are huge.
>> The entirety of the map retains 112MG.
>>
>> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
>> Looking at the "val" variable of a node, there is an
>> UpgradeProcessorInternal Inside this is a variable called
>> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I am looking at now retains 16MG of memory.
>> (Oddly, once I get this far, the retained sizes of its internal
>> objects don't really add up.)
>>
>> Any ideas on how to work around this? Or if this is already fixed in a later version of Tomcat?
>
> That isn't a leak. That is high memory usage. The objects will be GC'd once the HTTP/2 connection closes.
>
> The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.
>
> Mark
>
>
>>
>> Thanks,
>>
>> Mark Claassen
>> Senior Software Engineer
>>
>> Donnell Systems, Inc.
>> 130 South Main Street
>> Leighton Plaza Suite 375
>> South Bend, IN 46601
>> E-mail: mailto:mclaassen@ocie.net
>> Voice: (574)232-3784
>> Fax: (574)232-4014
>>
>> Disclaimer:
>> The opinions provided herein do not necessarily state or reflect those
>> of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes
>> no legal liability or responsibility for the posting.
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rob Sargent <rs...@xmission.com>
>> Sent: Thursday, July 8, 2021 6:50 PM
>> To: users@tomcat.apache.org
>> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
>> Importance: Low
>>
>>
>>
>> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>>> Ok. That didn’t seem to work. I will investigate further and try to find a way to send that information.
>>>
>>> It is not that busy a server, but the memory use increases very quickly. Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>>
>>> (We have a temporary monitor script in place that does a GC and then
>>> prints a class_histogram every half hour to help us pinpoint what is
>>> happening.)
>>>
>>> Thanks,
>>> Mark
>>>
>> Perhaps you've done this already, but
>>
>> grep -R 'static HashMap' src
>>
>> might locate some potential culprits.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thank you so much! This is good to know.
We may not change anything then (until that server gets upgraded). However, would switching back to HTTP 1.1 reduce the memory footprint in the mean time?
Would the connection be cleaned up when the browser goes away?
Mark Claassen
Senior Software Engineer
Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN 46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014
-------------------------------------------
Confidentiality Notice: OCIESERVICE
-------------------------------------------
The contents of this e-mail message and any attachments are intended solely for the addressee(s) named in this message. This communication is intended to be and to remain confidential. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not deliver, distribute, copy, disclose the contents or take any action in reliance upon the information contained in the communication or any attachments.
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Friday, July 9, 2021 12:59 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low
On 09/07/2021 16:21, Mark A. Claassen wrote:
> Thanks. I have done more heap analysis and think I have it tracked closer to the source.
>
> --
> I started looking at the heap a different way. The random values I looked at before (of the 80,000) may not have be as representative as I thought.
>
> Examining the retained sizes in the heap, I am finding:
> I have two instances of AbstractProtocol$ConnectionHandler.
>
> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
> This map as 32 elements in its "table". Most of these are null. Some of the ones that are not, are huge.
> The entirety of the map retains 112MG.
>
> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
> Looking at the "val" variable of a node, there is an
> UpgradeProcessorInternal Inside this is a variable called
> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I am looking at now retains 16MG of memory.
> (Oddly, once I get this far, the retained sizes of its internal
> objects don't really add up.)
>
> Any ideas on how to work around this? Or if this is already fixed in a later version of Tomcat?
That isn't a leak. That is high memory usage. The objects will be GC'd once the HTTP/2 connection closes.
The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.
Mark
>
> Thanks,
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN 46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those
> of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes
> no legal liability or responsibility for the posting.
>
>
>
>
> -----Original Message-----
> From: Rob Sargent <rs...@xmission.com>
> Sent: Thursday, July 8, 2021 6:50 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
>
>
>
> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>> Ok. That didn’t seem to work. I will investigate further and try to find a way to send that information.
>>
>> It is not that busy a server, but the memory use increases very quickly. Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>
>> (We have a temporary monitor script in place that does a GC and then
>> prints a class_histogram every half hour to help us pinpoint what is
>> happening.)
>>
>> Thanks,
>> Mark
>>
> Perhaps you've done this already, but
>
> grep -R 'static HashMap' src
>
> might locate some potential culprits.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by Mark Thomas <ma...@apache.org>.
On 09/07/2021 16:21, Mark A. Claassen wrote:
> Thanks. I have done more heap analysis and think I have it tracked closer to the source.
>
> --
> I started looking at the heap a different way. The random values I looked at before (of the 80,000) may not have be as representative as I thought.
>
> Examining the retained sizes in the heap, I am finding:
> I have two instances of AbstractProtocol$ConnectionHandler.
>
> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
> This map as 32 elements in its "table". Most of these are null. Some of the ones that are not, are huge.
> The entirety of the map retains 112MG.
>
> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
> Looking at the "val" variable of a node, there is an UpgradeProcessorInternal
> Inside this is a variable called internalHttpUpgradeHandler (of type Http2UpgradeHandler)
> The one of these I am looking at now retains 16MG of memory.
> (Oddly, once I get this far, the retained sizes of its internal objects don't really add up.)
>
> Any ideas on how to work around this? Or if this is already fixed in a later version of Tomcat?
That isn't a leak. That is high memory usage. The objects will be GC'd
once the HTTP/2 connection closes.
The changes in 9.0.37 / 9.0.39 should reduce the memory footprint
considerably.
Mark
>
> Thanks,
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN 46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect
> those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
> assumes no legal liability or responsibility for the posting.
>
>
>
>
> -----Original Message-----
> From: Rob Sargent <rs...@xmission.com>
> Sent: Thursday, July 8, 2021 6:50 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
>
>
>
> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>> Ok. That didn’t seem to work. I will investigate further and try to find a way to send that information.
>>
>> It is not that busy a server, but the memory use increases very quickly. Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>
>> (We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)
>>
>> Thanks,
>> Mark
>>
> Perhaps you've done this already, but
>
> grep -R 'static HashMap' src
>
> might locate some potential culprits.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thanks. I have done more heap analysis and think I have it tracked closer to the source.
--
I started looking at the heap a different way. The random values I looked at before (of the 80,000) may not have be as representative as I thought.
Examining the retained sizes in the heap, I am finding:
I have two instances of AbstractProtocol$ConnectionHandler.
One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
This map as 32 elements in its "table". Most of these are null. Some of the ones that are not, are huge.
The entirety of the map retains 112MG.
Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
Looking at the "val" variable of a node, there is an UpgradeProcessorInternal
Inside this is a variable called internalHttpUpgradeHandler (of type Http2UpgradeHandler)
The one of these I am looking at now retains 16MG of memory.
(Oddly, once I get this far, the retained sizes of its internal objects don't really add up.)
Any ideas on how to work around this? Or if this is already fixed in a later version of Tomcat?
Thanks,
Mark Claassen
Senior Software Engineer
Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN 46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014
Disclaimer:
The opinions provided herein do not necessarily state or reflect
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
assumes no legal liability or responsibility for the posting.
-----Original Message-----
From: Rob Sargent <rs...@xmission.com>
Sent: Thursday, July 8, 2021 6:50 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low
On 7/8/21 3:17 PM, Mark A. Claassen wrote:
> Ok. That didn’t seem to work. I will investigate further and try to find a way to send that information.
>
> It is not that busy a server, but the memory use increases very quickly. Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>
> (We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)
>
> Thanks,
> Mark
>
Perhaps you've done this already, but
grep -R 'static HashMap' src
might locate some potential culprits.
Re: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by Rob Sargent <rs...@xmission.com>.
On 7/8/21 3:17 PM, Mark A. Claassen wrote:
> Ok. That didn’t seem to work. I will investigate further and try to find a way to send that information.
>
> It is not that busy a server, but the memory use increases very quickly. Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>
> (We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)
>
> Thanks,
> Mark
>
Perhaps you've done this already, but
grep -R 'static HashMap' src
might locate some potential culprits.
RE: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by "Mark A. Claassen" <MC...@ocie.net>.
Ok. That didn’t seem to work. I will investigate further and try to find a way to send that information.
It is not that busy a server, but the memory use increases very quickly. Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
(We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)
Thanks,
Mark
From: Mark A. Claassen <MC...@ocie.net>
Sent: Thursday, July 8, 2021 5:07 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: RE: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low
Thanks for the prompt reply. The system was not that busy. Having over 80,000 seems wrong. I am going to try to send a picture; I am not sure if this will work. This is from the Netbeans profiler.
[cid:image001.png@01D7741B.B7CC7D90]
-----Original Message-----
From: Mark Thomas <ma...@apache.org>>
Sent: Thursday, July 8, 2021 2:46 PM
To: users@tomcat.apache.org<ma...@tomcat.apache.org>
Subject: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low
Memory leak, high memory usage or high GC churn?
The StreamProcessor shouldn't be a GC root. Either something should be retaining a reference to it or it should be eligible for GC.
There isn't much in the way of HTTP/2 specific leaks that have been fixed.
For HTTP/2, I'd expect you to see much more impact from the changes around 9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.
Mark
On 08/07/2021 19:32, Mark A. Claassen wrote:
> Sorry, realized I had a mistyped subject.
>
> -----Original Message-----
> From: Mark A. Claassen <MC...@ocie.net>>
> Sent: Thursday, July 8, 2021 2:30 PM
> To: Tomcat Users List <us...@tomcat.apache.org>>
> Subject: Http/s Memory Leak
> Importance: Low
>
> We are using 9.0.12 on a server and noticed a pretty big memory leak. The change log mentions a some fixed leaks in the releases since 9.0.12.
> I was wondering if this leak was fixed already.
>
> The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes. Below is the reference path.
> Is this one of the things fixed by setting useAsyncIO="false"? Is there a downside to using this?
> (We are using Http11AprProtocol and OpenSSL)
>
> Thanks,
> Mark
>
> Reference path of the leak:
> MessageBytes
> MimeHeaderField
> MimeHeaderField[]
> MimeHeaders
> Response
> Stream
> ConcurrentHashMap$Node
> ConcurrentHashMap$Node
> ConcurrentHashMap$Node[]
> Http2UpgradeHandler
> StreamProcessor (Root)
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN 46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>
For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>
RE: [Possible Spam] Re: HTTP/2 Memory Leak
Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thanks for the prompt reply. The system was not that busy. Having over 80,000 seems wrong. I am going to try to send a picture; I am not sure if this will work. This is from the Netbeans profiler.
[cid:image001.png@01D7741B.B7CC7D90]
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Thursday, July 8, 2021 2:46 PM
To: users@tomcat.apache.org
Subject: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low
Memory leak, high memory usage or high GC churn?
The StreamProcessor shouldn't be a GC root. Either something should be retaining a reference to it or it should be eligible for GC.
There isn't much in the way of HTTP/2 specific leaks that have been fixed.
For HTTP/2, I'd expect you to see much more impact from the changes around 9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.
Mark
On 08/07/2021 19:32, Mark A. Claassen wrote:
> Sorry, realized I had a mistyped subject.
>
> -----Original Message-----
> From: Mark A. Claassen <MC...@ocie.net>>
> Sent: Thursday, July 8, 2021 2:30 PM
> To: Tomcat Users List <us...@tomcat.apache.org>>
> Subject: Http/s Memory Leak
> Importance: Low
>
> We are using 9.0.12 on a server and noticed a pretty big memory leak. The change log mentions a some fixed leaks in the releases since 9.0.12.
> I was wondering if this leak was fixed already.
>
> The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes. Below is the reference path.
> Is this one of the things fixed by setting useAsyncIO="false"? Is there a downside to using this?
> (We are using Http11AprProtocol and OpenSSL)
>
> Thanks,
> Mark
>
> Reference path of the leak:
> MessageBytes
> MimeHeaderField
> MimeHeaderField[]
> MimeHeaders
> Response
> Stream
> ConcurrentHashMap$Node
> ConcurrentHashMap$Node
> ConcurrentHashMap$Node[]
> Http2UpgradeHandler
> StreamProcessor (Root)
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN 46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>
For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>
Re: HTTP/2 Memory Leak
Posted by Mark Thomas <ma...@apache.org>.
Memory leak, high memory usage or high GC churn?
The StreamProcessor shouldn't be a GC root. Either something should be
retaining a reference to it or it should be eligible for GC.
There isn't much in the way of HTTP/2 specific leaks that have been fixed.
For HTTP/2, I'd expect you to see much more impact from the changes
around 9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.
Mark
On 08/07/2021 19:32, Mark A. Claassen wrote:
> Sorry, realized I had a mistyped subject.
>
> -----Original Message-----
> From: Mark A. Claassen <MC...@ocie.net>
> Sent: Thursday, July 8, 2021 2:30 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Http/s Memory Leak
> Importance: Low
>
> We are using 9.0.12 on a server and noticed a pretty big memory leak. The change log mentions a some fixed leaks in the releases since 9.0.12.
> I was wondering if this leak was fixed already.
>
> The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes. Below is the reference path.
> Is this one of the things fixed by setting useAsyncIO="false"? Is there a downside to using this?
> (We are using Http11AprProtocol and OpenSSL)
>
> Thanks,
> Mark
>
> Reference path of the leak:
> MessageBytes
> MimeHeaderField
> MimeHeaderField[]
> MimeHeaders
> Response
> Stream
> ConcurrentHashMap$Node
> ConcurrentHashMap$Node
> ConcurrentHashMap$Node[]
> Http2UpgradeHandler
> StreamProcessor (Root)
>
> Mark Claassen
> Senior Software Engineer
>
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN 46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org