You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Mark A. Claassen" <MC...@ocie.net> on 2021/07/08 18:32:42 UTC

RE: HTTP/2 Memory Leak

Sorry, realized I had a mistyped subject.

-----Original Message-----
From: Mark A. Claassen <MC...@ocie.net> 
Sent: Thursday, July 8, 2021 2:30 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: Http/s Memory Leak
Importance: Low

We are using 9.0.12 on a server and noticed a pretty big memory leak.  The change log mentions a some fixed leaks in the releases since 9.0.12.  
I was wondering if this leak was fixed already.

The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.
Is this one of the things fixed by setting useAsyncIO="false"?  Is there a downside to using this?
(We are using Http11AprProtocol and OpenSSL)

Thanks,
Mark

Reference path of the leak:
	MessageBytes 
	MimeHeaderField
	MimeHeaderField[]
	MimeHeaders
	Response
	Stream
	ConcurrentHashMap$Node
	ConcurrentHashMap$Node
	ConcurrentHashMap$Node[]
	Http2UpgradeHandler
	StreamProcessor (Root)

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting. 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thanks for your help, and all the help you provide to people on this list.  Your replies are always quick, professional, and helpful.

A big part of what makes Tomcat so great is the assistance that can be found on this list!

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect 
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and 
assumes no legal liability or responsibility for the posting. 


-----Original Message-----
From: Mark Thomas <ma...@apache.org> 
Sent: Friday, July 9, 2021 1:31 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak

On 09/07/2021 18:23, Mark A. Claassen wrote:
> Thank you so much!  This is good to know.
> 
> We may not change anything then (until that server gets upgraded).  However, would switching back to HTTP 1.1 reduce the memory footprint in the mean time?

Yes.

The issue is the HTTP/2 priority tree. Essentially, Tomcat has to keep some information around about the previous few hundred requests on every
HTTP/2 connection. Until .37/.39 Tomcat just kept objects used for the request/response. As of .37/.39 Tomcat replaces these with a much lighter object once the request is completed.

> Would the connection be cleaned up when the browser goes away?

Should be.

Mark


> 
> Mark Claassen
> Senior Software Engineer
> 
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>    
> -------------------------------------------
> Confidentiality Notice: OCIESERVICE
> -------------------------------------------
> The contents of this e-mail message and any attachments are intended solely for the addressee(s) named in this message. This communication is intended to be and to remain confidential. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not deliver, distribute, copy, disclose the contents or take any action in reliance upon the information contained in the communication or any attachments.
> 
> 
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Friday, July 9, 2021 12:59 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
> 
> On 09/07/2021 16:21, Mark A. Claassen wrote:
>> Thanks.  I have done more heap analysis and think I have it tracked closer to the source.
>>
>> --
>> I started looking at the heap a different way.  The random values I looked at before (of the 80,000) may not have be as representative as I thought.
>>
>> Examining the retained sizes in the heap, I am finding:
>> I have two instances of AbstractProtocol$ConnectionHandler.
>>
>> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
>> This map as 32 elements in its "table".  Most of these are null.  Some of the ones that are not, are huge.
>> The entirety of the map retains 112MG.
>>
>> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
>> Looking at the "val" variable of a node, there is an 
>> UpgradeProcessorInternal Inside this is a variable called 
>> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I am looking at now retains 16MG of memory.
>> (Oddly, once I get this far, the retained sizes of its internal 
>> objects don't really add up.)
>>
>> Any ideas on how to work around this?  Or if this is already fixed in a later version of Tomcat?
> 
> That isn't a leak. That is high memory usage. The objects will be GC'd once the HTTP/2 connection closes.
> 
> The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.
> 
> Mark
> 
> 
>>
>> Thanks,
>>
>> Mark Claassen
>> Senior Software Engineer
>>
>> Donnell Systems, Inc.
>> 130 South Main Street
>> Leighton Plaza Suite 375
>> South Bend, IN  46601
>> E-mail: mailto:mclaassen@ocie.net
>> Voice: (574)232-3784
>> Fax: (574)232-4014
>>
>> Disclaimer:
>> The opinions provided herein do not necessarily state or reflect 
>> those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and 
>> assumes no legal liability or responsibility for the posting.
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rob Sargent <rs...@xmission.com>
>> Sent: Thursday, July 8, 2021 6:50 PM
>> To: users@tomcat.apache.org
>> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
>> Importance: Low
>>
>>
>>
>> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>>> Ok.  That didn’t seem to work.  I will investigate further and try to find a way to send that information.
>>>
>>> It is not that busy a server, but the memory use increases very quickly.  Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>>
>>> (We have a temporary monitor script in place that does a GC and then 
>>> prints a class_histogram every half hour to help us pinpoint what is
>>> happening.)
>>>
>>> Thanks,
>>> Mark
>>>
>> Perhaps you've done this already, but
>>
>>       grep -R 'static HashMap' src
>>
>> might locate some potential culprits.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by Mark Thomas <ma...@apache.org>.
On 09/07/2021 18:23, Mark A. Claassen wrote:
> Thank you so much!  This is good to know.
> 
> We may not change anything then (until that server gets upgraded).  However, would switching back to HTTP 1.1 reduce the memory footprint in the mean time?

Yes.

The issue is the HTTP/2 priority tree. Essentially, Tomcat has to keep 
some information around about the previous few hundred requests on every 
HTTP/2 connection. Until .37/.39 Tomcat just kept objects used for the 
request/response. As of .37/.39 Tomcat replaces these with a much 
lighter object once the request is completed.

> Would the connection be cleaned up when the browser goes away?

Should be.

Mark


> 
> Mark Claassen
> Senior Software Engineer
> 
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
>    
> -------------------------------------------
> Confidentiality Notice: OCIESERVICE
> -------------------------------------------
> The contents of this e-mail message and any attachments are intended solely for the addressee(s) named in this message. This communication is intended to be and to remain confidential. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not deliver, distribute, copy, disclose the contents or take any action in reliance upon the information contained in the communication or any attachments.
> 
> 
> -----Original Message-----
> From: Mark Thomas <ma...@apache.org>
> Sent: Friday, July 9, 2021 12:59 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
> 
> On 09/07/2021 16:21, Mark A. Claassen wrote:
>> Thanks.  I have done more heap analysis and think I have it tracked closer to the source.
>>
>> --
>> I started looking at the heap a different way.  The random values I looked at before (of the 80,000) may not have be as representative as I thought.
>>
>> Examining the retained sizes in the heap, I am finding:
>> I have two instances of AbstractProtocol$ConnectionHandler.
>>
>> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
>> This map as 32 elements in its "table".  Most of these are null.  Some of the ones that are not, are huge.
>> The entirety of the map retains 112MG.
>>
>> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
>> Looking at the "val" variable of a node, there is an
>> UpgradeProcessorInternal Inside this is a variable called
>> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I am looking at now retains 16MG of memory.
>> (Oddly, once I get this far, the retained sizes of its internal
>> objects don't really add up.)
>>
>> Any ideas on how to work around this?  Or if this is already fixed in a later version of Tomcat?
> 
> That isn't a leak. That is high memory usage. The objects will be GC'd once the HTTP/2 connection closes.
> 
> The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.
> 
> Mark
> 
> 
>>
>> Thanks,
>>
>> Mark Claassen
>> Senior Software Engineer
>>
>> Donnell Systems, Inc.
>> 130 South Main Street
>> Leighton Plaza Suite 375
>> South Bend, IN  46601
>> E-mail: mailto:mclaassen@ocie.net
>> Voice: (574)232-3784
>> Fax: (574)232-4014
>>
>> Disclaimer:
>> The opinions provided herein do not necessarily state or reflect those
>> of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes
>> no legal liability or responsibility for the posting.
>>
>>
>>
>>
>> -----Original Message-----
>> From: Rob Sargent <rs...@xmission.com>
>> Sent: Thursday, July 8, 2021 6:50 PM
>> To: users@tomcat.apache.org
>> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
>> Importance: Low
>>
>>
>>
>> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>>> Ok.  That didn’t seem to work.  I will investigate further and try to find a way to send that information.
>>>
>>> It is not that busy a server, but the memory use increases very quickly.  Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>>
>>> (We have a temporary monitor script in place that does a GC and then
>>> prints a class_histogram every half hour to help us pinpoint what is
>>> happening.)
>>>
>>> Thanks,
>>> Mark
>>>
>> Perhaps you've done this already, but
>>
>>       grep -R 'static HashMap' src
>>
>> might locate some potential culprits.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thank you so much!  This is good to know.

We may not change anything then (until that server gets upgraded).  However, would switching back to HTTP 1.1 reduce the memory footprint in the mean time?

Would the connection be cleaned up when the browser goes away?

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014
  
-------------------------------------------
Confidentiality Notice: OCIESERVICE
-------------------------------------------
The contents of this e-mail message and any attachments are intended solely for the addressee(s) named in this message. This communication is intended to be and to remain confidential. If you are not the intended recipient of this message, or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and its attachments. Do not deliver, distribute, copy, disclose the contents or take any action in reliance upon the information contained in the communication or any attachments.


-----Original Message-----
From: Mark Thomas <ma...@apache.org> 
Sent: Friday, July 9, 2021 12:59 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low

On 09/07/2021 16:21, Mark A. Claassen wrote:
> Thanks.  I have done more heap analysis and think I have it tracked closer to the source.
> 
> --
> I started looking at the heap a different way.  The random values I looked at before (of the 80,000) may not have be as representative as I thought.
> 
> Examining the retained sizes in the heap, I am finding:
> I have two instances of AbstractProtocol$ConnectionHandler.
> 
> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
> This map as 32 elements in its "table".  Most of these are null.  Some of the ones that are not, are huge.
> The entirety of the map retains 112MG.
> 
> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
> Looking at the "val" variable of a node, there is an 
> UpgradeProcessorInternal Inside this is a variable called 
> internalHttpUpgradeHandler (of type Http2UpgradeHandler) The one of these I am looking at now retains 16MG of memory.
> (Oddly, once I get this far, the retained sizes of its internal 
> objects don't really add up.)
> 
> Any ideas on how to work around this?  Or if this is already fixed in a later version of Tomcat?

That isn't a leak. That is high memory usage. The objects will be GC'd once the HTTP/2 connection closes.

The changes in 9.0.37 / 9.0.39 should reduce the memory footprint considerably.

Mark


> 
> Thanks,
> 
> Mark Claassen
> Senior Software Engineer
> 
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
> 
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those 
> of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes 
> no legal liability or responsibility for the posting.
> 
> 
> 
> 
> -----Original Message-----
> From: Rob Sargent <rs...@xmission.com>
> Sent: Thursday, July 8, 2021 6:50 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
> 
> 
> 
> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>> Ok.  That didn’t seem to work.  I will investigate further and try to find a way to send that information.
>>
>> It is not that busy a server, but the memory use increases very quickly.  Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>
>> (We have a temporary monitor script in place that does a GC and then 
>> prints a class_histogram every half hour to help us pinpoint what is 
>> happening.)
>>
>> Thanks,
>> Mark
>>
> Perhaps you've done this already, but
> 
>      grep -R 'static HashMap' src
> 
> might locate some potential culprits.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by Mark Thomas <ma...@apache.org>.
On 09/07/2021 16:21, Mark A. Claassen wrote:
> Thanks.  I have done more heap analysis and think I have it tracked closer to the source.
> 
> --
> I started looking at the heap a different way.  The random values I looked at before (of the 80,000) may not have be as representative as I thought.
> 
> Examining the retained sizes in the heap, I am finding:
> I have two instances of AbstractProtocol$ConnectionHandler.
> 
> One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
> This map as 32 elements in its "table".  Most of these are null.  Some of the ones that are not, are huge.
> The entirety of the map retains 112MG.
> 
> Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
> Looking at the "val" variable of a node, there is an UpgradeProcessorInternal
> Inside this is a variable called internalHttpUpgradeHandler (of type Http2UpgradeHandler)
> The one of these I am looking at now retains 16MG of memory.
> (Oddly, once I get this far, the retained sizes of its internal objects don't really add up.)
> 
> Any ideas on how to work around this?  Or if this is already fixed in a later version of Tomcat?

That isn't a leak. That is high memory usage. The objects will be GC'd 
once the HTTP/2 connection closes.

The changes in 9.0.37 / 9.0.39 should reduce the memory footprint 
considerably.

Mark


> 
> Thanks,
> 
> Mark Claassen
> Senior Software Engineer
> 
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
> 
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect
> those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and
> assumes no legal liability or responsibility for the posting.
> 
> 
> 
> 
> -----Original Message-----
> From: Rob Sargent <rs...@xmission.com>
> Sent: Thursday, July 8, 2021 6:50 PM
> To: users@tomcat.apache.org
> Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
> Importance: Low
> 
> 
> 
> On 7/8/21 3:17 PM, Mark A. Claassen wrote:
>> Ok.  That didn’t seem to work.  I will investigate further and try to find a way to send that information.
>>
>> It is not that busy a server, but the memory use increases very quickly.  Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>>
>> (We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)
>>
>> Thanks,
>> Mark
>>
> Perhaps you've done this already, but
> 
>      grep -R 'static HashMap' src
> 
> might locate some potential culprits.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thanks.  I have done more heap analysis and think I have it tracked closer to the source.

--
I started looking at the heap a different way.  The random values I looked at before (of the 80,000) may not have be as representative as I thought.

Examining the retained sizes in the heap, I am finding:
I have two instances of AbstractProtocol$ConnectionHandler.

One of these AbstractProtocol$ConnectionHandler instances has a ConcurrentHashMap called "connections"
This map as 32 elements in its "table".  Most of these are null.  Some of the ones that are not, are huge.
The entirety of the map retains 112MG.

Two of these ConcurrentHashMap$Node elements take up around 50MG a piece.
Looking at the "val" variable of a node, there is an UpgradeProcessorInternal
Inside this is a variable called internalHttpUpgradeHandler (of type Http2UpgradeHandler)
The one of these I am looking at now retains 16MG of memory.
(Oddly, once I get this far, the retained sizes of its internal objects don't really add up.)

Any ideas on how to work around this?  Or if this is already fixed in a later version of Tomcat?

Thanks,

Mark Claassen
Senior Software Engineer

Donnell Systems, Inc.
130 South Main Street
Leighton Plaza Suite 375
South Bend, IN  46601
E-mail: mailto:mclaassen@ocie.net
Voice: (574)232-3784
Fax: (574)232-4014

Disclaimer:
The opinions provided herein do not necessarily state or reflect 
those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and 
assumes no legal liability or responsibility for the posting. 




-----Original Message-----
From: Rob Sargent <rs...@xmission.com> 
Sent: Thursday, July 8, 2021 6:50 PM
To: users@tomcat.apache.org
Subject: Re: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low



On 7/8/21 3:17 PM, Mark A. Claassen wrote:
> Ok.  That didn’t seem to work.  I will investigate further and try to find a way to send that information.
>
> It is not that busy a server, but the memory use increases very quickly.  Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>
> (We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)
>
> Thanks,
> Mark
>
Perhaps you've done this already, but

    grep -R 'static HashMap' src

might locate some potential culprits.


Re: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by Rob Sargent <rs...@xmission.com>.

On 7/8/21 3:17 PM, Mark A. Claassen wrote:
> Ok.  That didn’t seem to work.  I will investigate further and try to find a way to send that information.
>
> It is not that busy a server, but the memory use increases very quickly.  Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.
>
> (We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)
>
> Thanks,
> Mark
>
Perhaps you've done this already, but

    grep -R 'static HashMap' src

might locate some potential culprits.


RE: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by "Mark A. Claassen" <MC...@ocie.net>.
Ok.  That didn’t seem to work.  I will investigate further and try to find a way to send that information.

It is not that busy a server, but the memory use increases very quickly.  Doing a class_histogram shows MessageBytes growing by the thousands every 30 minutes.

(We have a temporary monitor script in place that does a GC and then prints a class_histogram every half hour to help us pinpoint what is happening.)

Thanks,
Mark

From: Mark A. Claassen <MC...@ocie.net>
Sent: Thursday, July 8, 2021 5:07 PM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: RE: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low


Thanks for the prompt reply.  The system was not that busy.  Having over 80,000 seems wrong.  I am going to try to send a picture; I am not sure if this will work.  This is from the Netbeans profiler.



[cid:image001.png@01D7741B.B7CC7D90]



-----Original Message-----
From: Mark Thomas <ma...@apache.org>>
Sent: Thursday, July 8, 2021 2:46 PM
To: users@tomcat.apache.org<ma...@tomcat.apache.org>
Subject: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low



Memory leak, high memory usage or high GC churn?



The StreamProcessor shouldn't be a GC root. Either something should be retaining a reference to it or it should be eligible for GC.



There isn't much in the way of HTTP/2 specific leaks that have been fixed.



For HTTP/2, I'd expect you to see much more impact from the changes around 9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.



Mark





On 08/07/2021 19:32, Mark A. Claassen wrote:

> Sorry, realized I had a mistyped subject.

>

> -----Original Message-----

> From: Mark A. Claassen <MC...@ocie.net>>

> Sent: Thursday, July 8, 2021 2:30 PM

> To: Tomcat Users List <us...@tomcat.apache.org>>

> Subject: Http/s Memory Leak

> Importance: Low

>

> We are using 9.0.12 on a server and noticed a pretty big memory leak.  The change log mentions a some fixed leaks in the releases since 9.0.12.

> I was wondering if this leak was fixed already.

>

> The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.

> Is this one of the things fixed by setting useAsyncIO="false"?  Is there a downside to using this?

> (We are using Http11AprProtocol and OpenSSL)

>

> Thanks,

> Mark

>

> Reference path of the leak:

>     MessageBytes

>     MimeHeaderField

>     MimeHeaderField[]

>     MimeHeaders

>     Response

>     Stream

>     ConcurrentHashMap$Node

>     ConcurrentHashMap$Node

>     ConcurrentHashMap$Node[]

>     Http2UpgradeHandler

>     StreamProcessor (Root)

>

> Mark Claassen

> Senior Software Engineer

>

> Donnell Systems, Inc.

> 130 South Main Street

> Leighton Plaza Suite 375

> South Bend, IN  46601

> E-mail: mailto:mclaassen@ocie.net

> Voice: (574)232-3784

> Fax: (574)232-4014

>

> Disclaimer:

> The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting.

>

>

> ---------------------------------------------------------------------

> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>

> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>

>

>

> ---------------------------------------------------------------------

> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>

> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>

>





---------------------------------------------------------------------

To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>

For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>



RE: [Possible Spam] Re: HTTP/2 Memory Leak

Posted by "Mark A. Claassen" <MC...@ocie.net>.
Thanks for the prompt reply.  The system was not that busy.  Having over 80,000 seems wrong.  I am going to try to send a picture; I am not sure if this will work.  This is from the Netbeans profiler.



[cid:image001.png@01D7741B.B7CC7D90]



-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: Thursday, July 8, 2021 2:46 PM
To: users@tomcat.apache.org
Subject: [Possible Spam] Re: HTTP/2 Memory Leak
Importance: Low



Memory leak, high memory usage or high GC churn?



The StreamProcessor shouldn't be a GC root. Either something should be retaining a reference to it or it should be eligible for GC.



There isn't much in the way of HTTP/2 specific leaks that have been fixed.



For HTTP/2, I'd expect you to see much more impact from the changes around 9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.



Mark





On 08/07/2021 19:32, Mark A. Claassen wrote:

> Sorry, realized I had a mistyped subject.

>

> -----Original Message-----

> From: Mark A. Claassen <MC...@ocie.net>>

> Sent: Thursday, July 8, 2021 2:30 PM

> To: Tomcat Users List <us...@tomcat.apache.org>>

> Subject: Http/s Memory Leak

> Importance: Low

>

> We are using 9.0.12 on a server and noticed a pretty big memory leak.  The change log mentions a some fixed leaks in the releases since 9.0.12.

> I was wondering if this leak was fixed already.

>

> The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.

> Is this one of the things fixed by setting useAsyncIO="false"?  Is there a downside to using this?

> (We are using Http11AprProtocol and OpenSSL)

>

> Thanks,

> Mark

>

> Reference path of the leak:

>     MessageBytes

>     MimeHeaderField

>     MimeHeaderField[]

>     MimeHeaders

>     Response

>     Stream

>     ConcurrentHashMap$Node

>     ConcurrentHashMap$Node

>     ConcurrentHashMap$Node[]

>     Http2UpgradeHandler

>     StreamProcessor (Root)

>

> Mark Claassen

> Senior Software Engineer

>

> Donnell Systems, Inc.

> 130 South Main Street

> Leighton Plaza Suite 375

> South Bend, IN  46601

> E-mail: mailto:mclaassen@ocie.net

> Voice: (574)232-3784

> Fax: (574)232-4014

>

> Disclaimer:

> The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting.

>

>

> ---------------------------------------------------------------------

> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>

> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>

>

>

> ---------------------------------------------------------------------

> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>

> For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>

>





---------------------------------------------------------------------

To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>

For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>



Re: HTTP/2 Memory Leak

Posted by Mark Thomas <ma...@apache.org>.
Memory leak, high memory usage or high GC churn?

The StreamProcessor shouldn't be a GC root. Either something should be 
retaining a reference to it or it should be eligible for GC.

There isn't much in the way of HTTP/2 specific leaks that have been fixed.

For HTTP/2, I'd expect you to see much more impact from the changes 
around 9.0.37 / 9.0.39 that reduced the footprint of closed HTTP/2 streams.

Mark


On 08/07/2021 19:32, Mark A. Claassen wrote:
> Sorry, realized I had a mistyped subject.
> 
> -----Original Message-----
> From: Mark A. Claassen <MC...@ocie.net>
> Sent: Thursday, July 8, 2021 2:30 PM
> To: Tomcat Users List <us...@tomcat.apache.org>
> Subject: Http/s Memory Leak
> Importance: Low
> 
> We are using 9.0.12 on a server and noticed a pretty big memory leak.  The change log mentions a some fixed leaks in the releases since 9.0.12.
> I was wondering if this leak was fixed already.
> 
> The leak is that there are over 80,000 instances of org.apache.tomcat.util.buf.MessageBytes.  Below is the reference path.
> Is this one of the things fixed by setting useAsyncIO="false"?  Is there a downside to using this?
> (We are using Http11AprProtocol and OpenSSL)
> 
> Thanks,
> Mark
> 
> Reference path of the leak:
> 	MessageBytes
> 	MimeHeaderField
> 	MimeHeaderField[]
> 	MimeHeaders
> 	Response
> 	Stream
> 	ConcurrentHashMap$Node
> 	ConcurrentHashMap$Node
> 	ConcurrentHashMap$Node[]
> 	Http2UpgradeHandler
> 	StreamProcessor (Root)
> 
> Mark Claassen
> Senior Software Engineer
> 
> Donnell Systems, Inc.
> 130 South Main Street
> Leighton Plaza Suite 375
> South Bend, IN  46601
> E-mail: mailto:mclaassen@ocie.net
> Voice: (574)232-3784
> Fax: (574)232-4014
> 
> Disclaimer:
> The opinions provided herein do not necessarily state or reflect those of Donnell Systems, Inc.(DSI). DSI makes no warranty for and assumes no legal liability or responsibility for the posting.
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org