You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Chirag Dewan <ch...@gmail.com> on 2020/06/13 06:42:20 UTC
Connection Closure due to Fatal Stream with HTTP2
Hi,
We are observing that under high load, my clients start receiving a GoAway
frame with error:
*Connection[{id}], Stream[{id}] an error occurred during processing that
was fatal to the connection.*
Background : We have implemented our clients to close connections after
every 500-1000 requests (streams). This is a load balancer requirement that
we are working on and hence such a behavior. So with a throughput of around
19k, almost 40 connections are closed and recreated every second.
After we receive this frame, my clients start behaving erroneously. Before
this as well, my clients start sending RST_STREAM with canceled for each
request. Could this be due to the number of connections we open? Is it
related to the version of Tomcat? Or maybe my clients are misbehaving?
Now since I only receive this under heavy load, I can't quite picture
enough reasons for this to happen.
Any possible clues on where I should start looking?
My Stack:
Server - Tomcat 9.0.29
Client - Jetty 9.x
Thanks in advance
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Thanks a lot, Mark.
https://ibb.co/LgzFh6t - Memory snapshot after 15minutes of the test.
It's certainly better than the graph with 9.0.36, but I will wait for this
test to run for another few hours. I will update later.
Cheers,
Chirag
On Fri, Jun 26, 2020 at 6:20 PM Mark Thomas <ma...@apache.org> wrote:
> On 26/06/2020 12:48, Mark Thomas wrote:
> > On 26/06/2020 12:45, Chirag Dewan wrote:
> >> Absolutely Mark. Shouldn't take long.
> >
> > Great. I think I have found a potential root cause. If I am right, NIO
> > will show the same issues NIO2 did.
> >
> > I should have a test build for you shortly.
>
> Try this:
> https://people.apache.org/~markt/dev/v9.0.37-dev/
>
> Please note:
>
> This is NOT an official Apache Release.
>
> This build is only intended to be used to test this issue.
>
> Using the test build is it your own risk.
>
> Thanks,
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
On 26/06/2020 12:48, Mark Thomas wrote:
> On 26/06/2020 12:45, Chirag Dewan wrote:
>> Absolutely Mark. Shouldn't take long.
>
> Great. I think I have found a potential root cause. If I am right, NIO
> will show the same issues NIO2 did.
>
> I should have a test build for you shortly.
Try this:
https://people.apache.org/~markt/dev/v9.0.37-dev/
Please note:
This is NOT an official Apache Release.
This build is only intended to be used to test this issue.
Using the test build is it your own risk.
Thanks,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
On 26/06/2020 12:45, Chirag Dewan wrote:
> Absolutely Mark. Shouldn't take long.
Great. I think I have found a potential root cause. If I am right, NIO
will show the same issues NIO2 did.
I should have a test build for you shortly.
Mark
>
> On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas, <ma...@apache.org> wrote:
>
>> Aha!
>>
>> h2c could be the significant factor here. Let me take a look.
>>
>> Are you in a position to test against a dev build if the need arises?
>>
>> Mark
>>
>>
>> On 26/06/2020 11:30, Chirag Dewan wrote:
>>> Hey Mark,
>>>
>>> *Are you using h2c or h2 in your test?*
>>> We are using h2c
>>>
>>>
>>> *Do you see the same issue if you switch the the NIO connector? Note
>>> performance differences between NIO and NIO2 are very small.*
>>>
>>> I havent tried with NIO honestly. Can quickly execute and check. Will
>>> update the results.
>>>
>>> *How long does a single request take to process?*
>>> In normal scenarios, less than 3ms.
>>>
>>> Thanks,
>>> Chirag
>>>
>>> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <ma...@apache.org> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thanks for the additional information. The GC roots were particularly
>>>> informative.
>>>>
>>>> Those RequestInfo objects are associated with HTTP/1.1 requests, not
>>>> HTTP/2 requests.
>>>>
>>>> Some further questions to try and track down what is going on:
>>>>
>>>> - Are you using h2c or h2 in your test?
>>>>
>>>> - Do you see the same issue if you switch the the NIO connector? Note
>>>> performance differences between NIO and NIO2 are very small.
>>>>
>>>> - How long does a single request take to process?
>>>>
>>>> Thanks,
>>>>
>>>> Mark
>>>>
>>>> On 26/06/2020 09:24, Chirag Dewan wrote:
>>>>> Thanks Mark.
>>>>>
>>>>> *What is the typical response size for one of these requests? *
>>>>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
>>>>> response in my production use case, but the purpose here was to isolate
>>>> as
>>>>> many things as possible. Hence a dummy response.
>>>>>
>>>>> *How long does a typical test take to process? *
>>>>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
>> number
>>>>> of streams on my client was 500 - which means about 40 connections per
>>>>> second.
>>>>>
>>>>> * What are the GC roots for those RequestInfo objects?*
>>>>> https://ibb.co/fMRmCXZ
>>>>>
>>>>> I hope I was able to answer everything as expected. Thanks.
>>>>>
>>>>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> I've looked at the code and I have tried various tests but I am unable
>>>>>> to re-create a memory leak.
>>>>>>
>>>>>> The code used to (before I made a few changes this afternoon) retain a
>>>>>> lot more memory per Stream and it is possible that what you are seeing
>>>>>> is a system that doesn't have enough memory to achieve steady state.
>>>>>>
>>>>>> If you are able to build the latest 9.0.x and test that, that could be
>>>>>> helpful. Alternatively, I could provide a test build for you to
>>>>>> experiment with.
>>>>>>
>>>>>> Some additional questions that might aid understanding:
>>>>>>
>>>>>> - What is the typical response size for one of these requests?
>>>>>> - How long does a typical test take to process?
>>>>>> - What are the GC roots for those RequestInfo objects?
>>>>>>
>>>>>> Thanks again,
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>>>>>> Hi Mark,
>>>>>>>
>>>>>>> Its the default APR connector with 150 Threads.
>>>>>>>
>>>>>>> Chirag
>>>>>>>
>>>>>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
>>>>>>>
>>>>>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>>>>>>>> Thanks for the quick check Mark.
>>>>>>>>>
>>>>>>>>> These are the images I tried referring to:
>>>>>>>>>
>>>>>>>>> https://ibb.co/LzKtRgh
>>>>>>>>>
>>>>>>>>> https://ibb.co/2s7hqRL
>>>>>>>>>
>>>>>>>>> https://ibb.co/KmKj590
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> The last one is the MAT screenshot showing many RequestInfo
>> objects.
>>>>>>>>
>>>>>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
>>>>>>>> look. Out of interest, how many threads is the Connector configured
>> to
>>>>>> use?
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Chirag
>>>>>>>>>
>>>>>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org>
>>>> wrote:
>>>>>>>>>
>>>>>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
>>>> opens
>>>>>>>> new
>>>>>>>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>>>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>>>>>>>
>>>>>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I
>> get a
>>>>>>>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>>>>>>>
>>>>>>>>>>>> For better understanding, let me explain my use case again:
>>>>>>>>>>>>
>>>>>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>>>>>>>> requirement is to close a connection after a configurable(say
>>>> 5000)
>>>>>>>>>>>> number of requests/streams and open a new connection that
>>>> continues
>>>>>> to
>>>>>>>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>>>>>>>
>>>>>>>>>>>> When I execute this use case under load, I see that after
>> ~2hours
>>>> my
>>>>>>>>>>>> requests fail and I get a series of errors like request
>>>>>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
>>>>>> close
>>>>>>>>>>>> exception on my client.
>>>>>>>>>>>> On further debugging, I found that it's a Tomcat memory problem
>>>> and
>>>>>> it
>>>>>>>>>>>> goes OOM after sometime under heavy load with multiple
>> connections
>>>>>>>> being
>>>>>>>>>>>> re-established by the clients.
>>>>>>>>>>>>
>>>>>>>>>>>> image.png
>>>>>>>>>>>>
>>>>>>>>>>>> image.png
>>>>>>>>>>>>
>>>>>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>>>>>>>
>>>>>>>>>>> Embedded images get dropped by the list software. Post those
>> images
>>>>>>>>>>> somewhere we can see them.
>>>>>>>>>>>
>>>>>>>>>>>> Please let me know if you any experience with such a situation.
>>>>>> Thanks
>>>>>>>>>>>> in advance.
>>>>>>>>>>>
>>>>>>>>>>> Nothing comes to mind.
>>>>>>>>>>>
>>>>>>>>>>> I'll try some simple tests with HTTP/2.
>>>>>>>>>>
>>>>>>>>>> I don't see a memory leak (the memory is reclaimed eventually)
>> but I
>>>>>> do
>>>>>>>>>> see possibilities to release memory associated with request
>>>> processing
>>>>>>>>>> sooner.
>>>>>>>>>>
>>>>>>>>>> Right now you need to allocate more memory to the Java process to
>>>>>> enable
>>>>>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>>>>>>>
>>>>>>>>>> It looks like a reasonable chunk of memory is released when the
>>>>>>>>>> Connection closes that could be released earlier when the
>> associated
>>>>>>>>>> Stream closes. I'll take a look at what can be done in that area.
>> In
>>>>>> the
>>>>>>>>>> meantime, reducing the number of Streams you allow on a Connection
>>>>>>>>>> before it is closed should reduce overall memory usage.
>>>>>>>>>>
>>>>>>>>>> Mark
>>>>>>>>>>
>>>>>>>>>>
>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Absolutely Mark. Shouldn't take long.
On Fri, 26 Jun, 2020, 4:16 pm Mark Thomas, <ma...@apache.org> wrote:
> Aha!
>
> h2c could be the significant factor here. Let me take a look.
>
> Are you in a position to test against a dev build if the need arises?
>
> Mark
>
>
> On 26/06/2020 11:30, Chirag Dewan wrote:
> > Hey Mark,
> >
> > *Are you using h2c or h2 in your test?*
> > We are using h2c
> >
> >
> > *Do you see the same issue if you switch the the NIO connector? Note
> > performance differences between NIO and NIO2 are very small.*
> >
> > I havent tried with NIO honestly. Can quickly execute and check. Will
> > update the results.
> >
> > *How long does a single request take to process?*
> > In normal scenarios, less than 3ms.
> >
> > Thanks,
> > Chirag
> >
> > On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <ma...@apache.org> wrote:
> >
> >> Hi,
> >>
> >> Thanks for the additional information. The GC roots were particularly
> >> informative.
> >>
> >> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> >> HTTP/2 requests.
> >>
> >> Some further questions to try and track down what is going on:
> >>
> >> - Are you using h2c or h2 in your test?
> >>
> >> - Do you see the same issue if you switch the the NIO connector? Note
> >> performance differences between NIO and NIO2 are very small.
> >>
> >> - How long does a single request take to process?
> >>
> >> Thanks,
> >>
> >> Mark
> >>
> >> On 26/06/2020 09:24, Chirag Dewan wrote:
> >>> Thanks Mark.
> >>>
> >>> *What is the typical response size for one of these requests? *
> >>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> >>> response in my production use case, but the purpose here was to isolate
> >> as
> >>> many things as possible. Hence a dummy response.
> >>>
> >>> *How long does a typical test take to process? *
> >>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The
> number
> >>> of streams on my client was 500 - which means about 40 connections per
> >>> second.
> >>>
> >>> * What are the GC roots for those RequestInfo objects?*
> >>> https://ibb.co/fMRmCXZ
> >>>
> >>> I hope I was able to answer everything as expected. Thanks.
> >>>
> >>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
> >>>
> >>>> Thanks.
> >>>>
> >>>> I've looked at the code and I have tried various tests but I am unable
> >>>> to re-create a memory leak.
> >>>>
> >>>> The code used to (before I made a few changes this afternoon) retain a
> >>>> lot more memory per Stream and it is possible that what you are seeing
> >>>> is a system that doesn't have enough memory to achieve steady state.
> >>>>
> >>>> If you are able to build the latest 9.0.x and test that, that could be
> >>>> helpful. Alternatively, I could provide a test build for you to
> >>>> experiment with.
> >>>>
> >>>> Some additional questions that might aid understanding:
> >>>>
> >>>> - What is the typical response size for one of these requests?
> >>>> - How long does a typical test take to process?
> >>>> - What are the GC roots for those RequestInfo objects?
> >>>>
> >>>> Thanks again,
> >>>>
> >>>> Mark
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 25/06/2020 15:10, Chirag Dewan wrote:
> >>>>> Hi Mark,
> >>>>>
> >>>>> Its the default APR connector with 150 Threads.
> >>>>>
> >>>>> Chirag
> >>>>>
> >>>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
> >>>>>
> >>>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>>>>>> Thanks for the quick check Mark.
> >>>>>>>
> >>>>>>> These are the images I tried referring to:
> >>>>>>>
> >>>>>>> https://ibb.co/LzKtRgh
> >>>>>>>
> >>>>>>> https://ibb.co/2s7hqRL
> >>>>>>>
> >>>>>>> https://ibb.co/KmKj590
> >>>>>>>
> >>>>>>>
> >>>>>>> The last one is the MAT screenshot showing many RequestInfo
> objects.
> >>>>>>
> >>>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
> >>>>>> look. Out of interest, how many threads is the Connector configured
> to
> >>>> use?
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Chirag
> >>>>>>>
> >>>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org>
> >> wrote:
> >>>>>>>
> >>>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
> >>>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>>>
> >>>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
> >> opens
> >>>>>> new
> >>>>>>>>>> connections every second. In the memory dump, we see a lot of
> >>>>>>>>>> RequestInfo objects that are causing the memory spike.
> >>>>>>>>>>
> >>>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I
> get a
> >>>>>>>>>> request timed out on my client). This seems like a bug to me.
> >>>>>>>>>>
> >>>>>>>>>> For better understanding, let me explain my use case again:
> >>>>>>>>>>
> >>>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>>>>>>>> requirement is to close a connection after a configurable(say
> >> 5000)
> >>>>>>>>>> number of requests/streams and open a new connection that
> >> continues
> >>>> to
> >>>>>>>>>> send requests. I close a connection by sending a GoAway frame.
> >>>>>>>>>>
> >>>>>>>>>> When I execute this use case under load, I see that after
> ~2hours
> >> my
> >>>>>>>>>> requests fail and I get a series of errors like request
> >>>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
> >>>> close
> >>>>>>>>>> exception on my client.
> >>>>>>>>>> On further debugging, I found that it's a Tomcat memory problem
> >> and
> >>>> it
> >>>>>>>>>> goes OOM after sometime under heavy load with multiple
> connections
> >>>>>> being
> >>>>>>>>>> re-established by the clients.
> >>>>>>>>>>
> >>>>>>>>>> image.png
> >>>>>>>>>>
> >>>>>>>>>> image.png
> >>>>>>>>>>
> >>>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>>>>>>>
> >>>>>>>>> Embedded images get dropped by the list software. Post those
> images
> >>>>>>>>> somewhere we can see them.
> >>>>>>>>>
> >>>>>>>>>> Please let me know if you any experience with such a situation.
> >>>> Thanks
> >>>>>>>>>> in advance.
> >>>>>>>>>
> >>>>>>>>> Nothing comes to mind.
> >>>>>>>>>
> >>>>>>>>> I'll try some simple tests with HTTP/2.
> >>>>>>>>
> >>>>>>>> I don't see a memory leak (the memory is reclaimed eventually)
> but I
> >>>> do
> >>>>>>>> see possibilities to release memory associated with request
> >> processing
> >>>>>>>> sooner.
> >>>>>>>>
> >>>>>>>> Right now you need to allocate more memory to the Java process to
> >>>> enable
> >>>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
> >>>>>>>>
> >>>>>>>> It looks like a reasonable chunk of memory is released when the
> >>>>>>>> Connection closes that could be released earlier when the
> associated
> >>>>>>>> Stream closes. I'll take a look at what can be done in that area.
> In
> >>>> the
> >>>>>>>> meantime, reducing the number of Streams you allow on a Connection
> >>>>>>>> before it is closed should reduce overall memory usage.
> >>>>>>>>
> >>>>>>>> Mark
> >>>>>>>>
> >>>>>>>>
> >> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
Aha!
h2c could be the significant factor here. Let me take a look.
Are you in a position to test against a dev build if the need arises?
Mark
On 26/06/2020 11:30, Chirag Dewan wrote:
> Hey Mark,
>
> *Are you using h2c or h2 in your test?*
> We are using h2c
>
>
> *Do you see the same issue if you switch the the NIO connector? Note
> performance differences between NIO and NIO2 are very small.*
>
> I havent tried with NIO honestly. Can quickly execute and check. Will
> update the results.
>
> *How long does a single request take to process?*
> In normal scenarios, less than 3ms.
>
> Thanks,
> Chirag
>
> On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <ma...@apache.org> wrote:
>
>> Hi,
>>
>> Thanks for the additional information. The GC roots were particularly
>> informative.
>>
>> Those RequestInfo objects are associated with HTTP/1.1 requests, not
>> HTTP/2 requests.
>>
>> Some further questions to try and track down what is going on:
>>
>> - Are you using h2c or h2 in your test?
>>
>> - Do you see the same issue if you switch the the NIO connector? Note
>> performance differences between NIO and NIO2 are very small.
>>
>> - How long does a single request take to process?
>>
>> Thanks,
>>
>> Mark
>>
>> On 26/06/2020 09:24, Chirag Dewan wrote:
>>> Thanks Mark.
>>>
>>> *What is the typical response size for one of these requests? *
>>> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
>>> response in my production use case, but the purpose here was to isolate
>> as
>>> many things as possible. Hence a dummy response.
>>>
>>> *How long does a typical test take to process? *
>>> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
>>> of streams on my client was 500 - which means about 40 connections per
>>> second.
>>>
>>> * What are the GC roots for those RequestInfo objects?*
>>> https://ibb.co/fMRmCXZ
>>>
>>> I hope I was able to answer everything as expected. Thanks.
>>>
>>> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
>>>
>>>> Thanks.
>>>>
>>>> I've looked at the code and I have tried various tests but I am unable
>>>> to re-create a memory leak.
>>>>
>>>> The code used to (before I made a few changes this afternoon) retain a
>>>> lot more memory per Stream and it is possible that what you are seeing
>>>> is a system that doesn't have enough memory to achieve steady state.
>>>>
>>>> If you are able to build the latest 9.0.x and test that, that could be
>>>> helpful. Alternatively, I could provide a test build for you to
>>>> experiment with.
>>>>
>>>> Some additional questions that might aid understanding:
>>>>
>>>> - What is the typical response size for one of these requests?
>>>> - How long does a typical test take to process?
>>>> - What are the GC roots for those RequestInfo objects?
>>>>
>>>> Thanks again,
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>>>> Hi Mark,
>>>>>
>>>>> Its the default APR connector with 150 Threads.
>>>>>
>>>>> Chirag
>>>>>
>>>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
>>>>>
>>>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>>>>>> Thanks for the quick check Mark.
>>>>>>>
>>>>>>> These are the images I tried referring to:
>>>>>>>
>>>>>>> https://ibb.co/LzKtRgh
>>>>>>>
>>>>>>> https://ibb.co/2s7hqRL
>>>>>>>
>>>>>>> https://ibb.co/KmKj590
>>>>>>>
>>>>>>>
>>>>>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>>>>>
>>>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
>>>>>> look. Out of interest, how many threads is the Connector configured to
>>>> use?
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Chirag
>>>>>>>
>>>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org>
>> wrote:
>>>>>>>
>>>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
>> opens
>>>>>> new
>>>>>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>>>>>
>>>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>>>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>>>>>
>>>>>>>>>> For better understanding, let me explain my use case again:
>>>>>>>>>>
>>>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>>>>>> requirement is to close a connection after a configurable(say
>> 5000)
>>>>>>>>>> number of requests/streams and open a new connection that
>> continues
>>>> to
>>>>>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>>>>>
>>>>>>>>>> When I execute this use case under load, I see that after ~2hours
>> my
>>>>>>>>>> requests fail and I get a series of errors like request
>>>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
>>>> close
>>>>>>>>>> exception on my client.
>>>>>>>>>> On further debugging, I found that it's a Tomcat memory problem
>> and
>>>> it
>>>>>>>>>> goes OOM after sometime under heavy load with multiple connections
>>>>>> being
>>>>>>>>>> re-established by the clients.
>>>>>>>>>>
>>>>>>>>>> image.png
>>>>>>>>>>
>>>>>>>>>> image.png
>>>>>>>>>>
>>>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>>>>>
>>>>>>>>> Embedded images get dropped by the list software. Post those images
>>>>>>>>> somewhere we can see them.
>>>>>>>>>
>>>>>>>>>> Please let me know if you any experience with such a situation.
>>>> Thanks
>>>>>>>>>> in advance.
>>>>>>>>>
>>>>>>>>> Nothing comes to mind.
>>>>>>>>>
>>>>>>>>> I'll try some simple tests with HTTP/2.
>>>>>>>>
>>>>>>>> I don't see a memory leak (the memory is reclaimed eventually) but I
>>>> do
>>>>>>>> see possibilities to release memory associated with request
>> processing
>>>>>>>> sooner.
>>>>>>>>
>>>>>>>> Right now you need to allocate more memory to the Java process to
>>>> enable
>>>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>>>>>
>>>>>>>> It looks like a reasonable chunk of memory is released when the
>>>>>>>> Connection closes that could be released earlier when the associated
>>>>>>>> Stream closes. I'll take a look at what can be done in that area. In
>>>> the
>>>>>>>> meantime, reducing the number of Streams you allow on a Connection
>>>>>>>> before it is closed should reduce overall memory usage.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>>
>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Hey Mark,
*Are you using h2c or h2 in your test?*
We are using h2c
*Do you see the same issue if you switch the the NIO connector? Note
performance differences between NIO and NIO2 are very small.*
I havent tried with NIO honestly. Can quickly execute and check. Will
update the results.
*How long does a single request take to process?*
In normal scenarios, less than 3ms.
Thanks,
Chirag
On Fri, Jun 26, 2020 at 3:26 PM Mark Thomas <ma...@apache.org> wrote:
> Hi,
>
> Thanks for the additional information. The GC roots were particularly
> informative.
>
> Those RequestInfo objects are associated with HTTP/1.1 requests, not
> HTTP/2 requests.
>
> Some further questions to try and track down what is going on:
>
> - Are you using h2c or h2 in your test?
>
> - Do you see the same issue if you switch the the NIO connector? Note
> performance differences between NIO and NIO2 are very small.
>
> - How long does a single request take to process?
>
> Thanks,
>
> Mark
>
> On 26/06/2020 09:24, Chirag Dewan wrote:
> > Thanks Mark.
> >
> > *What is the typical response size for one of these requests? *
> > It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> > response in my production use case, but the purpose here was to isolate
> as
> > many things as possible. Hence a dummy response.
> >
> > *How long does a typical test take to process? *
> > I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> > of streams on my client was 500 - which means about 40 connections per
> > second.
> >
> > * What are the GC roots for those RequestInfo objects?*
> > https://ibb.co/fMRmCXZ
> >
> > I hope I was able to answer everything as expected. Thanks.
> >
> > On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
> >
> >> Thanks.
> >>
> >> I've looked at the code and I have tried various tests but I am unable
> >> to re-create a memory leak.
> >>
> >> The code used to (before I made a few changes this afternoon) retain a
> >> lot more memory per Stream and it is possible that what you are seeing
> >> is a system that doesn't have enough memory to achieve steady state.
> >>
> >> If you are able to build the latest 9.0.x and test that, that could be
> >> helpful. Alternatively, I could provide a test build for you to
> >> experiment with.
> >>
> >> Some additional questions that might aid understanding:
> >>
> >> - What is the typical response size for one of these requests?
> >> - How long does a typical test take to process?
> >> - What are the GC roots for those RequestInfo objects?
> >>
> >> Thanks again,
> >>
> >> Mark
> >>
> >>
> >>
> >>
> >> On 25/06/2020 15:10, Chirag Dewan wrote:
> >>> Hi Mark,
> >>>
> >>> Its the default APR connector with 150 Threads.
> >>>
> >>> Chirag
> >>>
> >>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
> >>>
> >>>> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>>>> Thanks for the quick check Mark.
> >>>>>
> >>>>> These are the images I tried referring to:
> >>>>>
> >>>>> https://ibb.co/LzKtRgh
> >>>>>
> >>>>> https://ibb.co/2s7hqRL
> >>>>>
> >>>>> https://ibb.co/KmKj590
> >>>>>
> >>>>>
> >>>>> The last one is the MAT screenshot showing many RequestInfo objects.
> >>>>
> >>>> Thanks. That certainly looks like a memory leak. I'll take a closer
> >>>> look. Out of interest, how many threads is the Connector configured to
> >> use?
> >>>>
> >>>> Mark
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Chirag
> >>>>>
> >>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org>
> wrote:
> >>>>>
> >>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
> >>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Update: We found that Tomcat goes OOM when a client closes and
> opens
> >>>> new
> >>>>>>>> connections every second. In the memory dump, we see a lot of
> >>>>>>>> RequestInfo objects that are causing the memory spike.
> >>>>>>>>
> >>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
> >>>>>>>> request timed out on my client). This seems like a bug to me.
> >>>>>>>>
> >>>>>>>> For better understanding, let me explain my use case again:
> >>>>>>>>
> >>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>>>>>> requirement is to close a connection after a configurable(say
> 5000)
> >>>>>>>> number of requests/streams and open a new connection that
> continues
> >> to
> >>>>>>>> send requests. I close a connection by sending a GoAway frame.
> >>>>>>>>
> >>>>>>>> When I execute this use case under load, I see that after ~2hours
> my
> >>>>>>>> requests fail and I get a series of errors like request
> >>>>>>>> timeouts(5seconds), invalid window update frame, and connection
> >> close
> >>>>>>>> exception on my client.
> >>>>>>>> On further debugging, I found that it's a Tomcat memory problem
> and
> >> it
> >>>>>>>> goes OOM after sometime under heavy load with multiple connections
> >>>> being
> >>>>>>>> re-established by the clients.
> >>>>>>>>
> >>>>>>>> image.png
> >>>>>>>>
> >>>>>>>> image.png
> >>>>>>>>
> >>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>>>>>
> >>>>>>> Embedded images get dropped by the list software. Post those images
> >>>>>>> somewhere we can see them.
> >>>>>>>
> >>>>>>>> Please let me know if you any experience with such a situation.
> >> Thanks
> >>>>>>>> in advance.
> >>>>>>>
> >>>>>>> Nothing comes to mind.
> >>>>>>>
> >>>>>>> I'll try some simple tests with HTTP/2.
> >>>>>>
> >>>>>> I don't see a memory leak (the memory is reclaimed eventually) but I
> >> do
> >>>>>> see possibilities to release memory associated with request
> processing
> >>>>>> sooner.
> >>>>>>
> >>>>>> Right now you need to allocate more memory to the Java process to
> >> enable
> >>>>>> Tomcat to handle the HTTP/2 load it is presented with.
> >>>>>>
> >>>>>> It looks like a reasonable chunk of memory is released when the
> >>>>>> Connection closes that could be released earlier when the associated
> >>>>>> Stream closes. I'll take a look at what can be done in that area. In
> >> the
> >>>>>> meantime, reducing the number of Streams you allow on a Connection
> >>>>>> before it is closed should reduce overall memory usage.
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
Hi,
Thanks for the additional information. The GC roots were particularly
informative.
Those RequestInfo objects are associated with HTTP/1.1 requests, not
HTTP/2 requests.
Some further questions to try and track down what is going on:
- Are you using h2c or h2 in your test?
- Do you see the same issue if you switch the the NIO connector? Note
performance differences between NIO and NIO2 are very small.
- How long does a single request take to process?
Thanks,
Mark
On 26/06/2020 09:24, Chirag Dewan wrote:
> Thanks Mark.
>
> *What is the typical response size for one of these requests? *
> It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
> response in my production use case, but the purpose here was to isolate as
> many things as possible. Hence a dummy response.
>
> *How long does a typical test take to process? *
> I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
> of streams on my client was 500 - which means about 40 connections per
> second.
>
> * What are the GC roots for those RequestInfo objects?*
> https://ibb.co/fMRmCXZ
>
> I hope I was able to answer everything as expected. Thanks.
>
> On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
>
>> Thanks.
>>
>> I've looked at the code and I have tried various tests but I am unable
>> to re-create a memory leak.
>>
>> The code used to (before I made a few changes this afternoon) retain a
>> lot more memory per Stream and it is possible that what you are seeing
>> is a system that doesn't have enough memory to achieve steady state.
>>
>> If you are able to build the latest 9.0.x and test that, that could be
>> helpful. Alternatively, I could provide a test build for you to
>> experiment with.
>>
>> Some additional questions that might aid understanding:
>>
>> - What is the typical response size for one of these requests?
>> - How long does a typical test take to process?
>> - What are the GC roots for those RequestInfo objects?
>>
>> Thanks again,
>>
>> Mark
>>
>>
>>
>>
>> On 25/06/2020 15:10, Chirag Dewan wrote:
>>> Hi Mark,
>>>
>>> Its the default APR connector with 150 Threads.
>>>
>>> Chirag
>>>
>>> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
>>>
>>>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>>>> Thanks for the quick check Mark.
>>>>>
>>>>> These are the images I tried referring to:
>>>>>
>>>>> https://ibb.co/LzKtRgh
>>>>>
>>>>> https://ibb.co/2s7hqRL
>>>>>
>>>>> https://ibb.co/KmKj590
>>>>>
>>>>>
>>>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>>>
>>>> Thanks. That certainly looks like a memory leak. I'll take a closer
>>>> look. Out of interest, how many threads is the Connector configured to
>> use?
>>>>
>>>> Mark
>>>>
>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Chirag
>>>>>
>>>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
>>>>>
>>>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Update: We found that Tomcat goes OOM when a client closes and opens
>>>> new
>>>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>>>
>>>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>>>
>>>>>>>> For better understanding, let me explain my use case again:
>>>>>>>>
>>>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>>>> requirement is to close a connection after a configurable(say 5000)
>>>>>>>> number of requests/streams and open a new connection that continues
>> to
>>>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>>>
>>>>>>>> When I execute this use case under load, I see that after ~2hours my
>>>>>>>> requests fail and I get a series of errors like request
>>>>>>>> timeouts(5seconds), invalid window update frame, and connection
>> close
>>>>>>>> exception on my client.
>>>>>>>> On further debugging, I found that it's a Tomcat memory problem and
>> it
>>>>>>>> goes OOM after sometime under heavy load with multiple connections
>>>> being
>>>>>>>> re-established by the clients.
>>>>>>>>
>>>>>>>> image.png
>>>>>>>>
>>>>>>>> image.png
>>>>>>>>
>>>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>>>
>>>>>>> Embedded images get dropped by the list software. Post those images
>>>>>>> somewhere we can see them.
>>>>>>>
>>>>>>>> Please let me know if you any experience with such a situation.
>> Thanks
>>>>>>>> in advance.
>>>>>>>
>>>>>>> Nothing comes to mind.
>>>>>>>
>>>>>>> I'll try some simple tests with HTTP/2.
>>>>>>
>>>>>> I don't see a memory leak (the memory is reclaimed eventually) but I
>> do
>>>>>> see possibilities to release memory associated with request processing
>>>>>> sooner.
>>>>>>
>>>>>> Right now you need to allocate more memory to the Java process to
>> enable
>>>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>>>
>>>>>> It looks like a reasonable chunk of memory is released when the
>>>>>> Connection closes that could be released earlier when the associated
>>>>>> Stream closes. I'll take a look at what can be done in that area. In
>> the
>>>>>> meantime, reducing the number of Streams you allow on a Connection
>>>>>> before it is closed should reduce overall memory usage.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Thanks Mark.
*What is the typical response size for one of these requests? *
It' basically a dummy response JSON of ~300bytes. I expect 2300bytes of
response in my production use case, but the purpose here was to isolate as
many things as possible. Hence a dummy response.
*How long does a typical test take to process? *
I see Tomcat's memory reaching 28G in about 2hours at 19K TPS. The number
of streams on my client was 500 - which means about 40 connections per
second.
* What are the GC roots for those RequestInfo objects?*
https://ibb.co/fMRmCXZ
I hope I was able to answer everything as expected. Thanks.
On Thu, Jun 25, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
> Thanks.
>
> I've looked at the code and I have tried various tests but I am unable
> to re-create a memory leak.
>
> The code used to (before I made a few changes this afternoon) retain a
> lot more memory per Stream and it is possible that what you are seeing
> is a system that doesn't have enough memory to achieve steady state.
>
> If you are able to build the latest 9.0.x and test that, that could be
> helpful. Alternatively, I could provide a test build for you to
> experiment with.
>
> Some additional questions that might aid understanding:
>
> - What is the typical response size for one of these requests?
> - How long does a typical test take to process?
> - What are the GC roots for those RequestInfo objects?
>
> Thanks again,
>
> Mark
>
>
>
>
> On 25/06/2020 15:10, Chirag Dewan wrote:
> > Hi Mark,
> >
> > Its the default APR connector with 150 Threads.
> >
> > Chirag
> >
> > On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
> >
> >> On 25/06/2020 11:00, Chirag Dewan wrote:
> >>> Thanks for the quick check Mark.
> >>>
> >>> These are the images I tried referring to:
> >>>
> >>> https://ibb.co/LzKtRgh
> >>>
> >>> https://ibb.co/2s7hqRL
> >>>
> >>> https://ibb.co/KmKj590
> >>>
> >>>
> >>> The last one is the MAT screenshot showing many RequestInfo objects.
> >>
> >> Thanks. That certainly looks like a memory leak. I'll take a closer
> >> look. Out of interest, how many threads is the Connector configured to
> use?
> >>
> >> Mark
> >>
> >>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Chirag
> >>>
> >>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
> >>>
> >>>> On 24/06/2020 12:17, Mark Thomas wrote:
> >>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> Update: We found that Tomcat goes OOM when a client closes and opens
> >> new
> >>>>>> connections every second. In the memory dump, we see a lot of
> >>>>>> RequestInfo objects that are causing the memory spike.
> >>>>>>
> >>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
> >>>>>> request timed out on my client). This seems like a bug to me.
> >>>>>>
> >>>>>> For better understanding, let me explain my use case again:
> >>>>>>
> >>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>>>> requirement is to close a connection after a configurable(say 5000)
> >>>>>> number of requests/streams and open a new connection that continues
> to
> >>>>>> send requests. I close a connection by sending a GoAway frame.
> >>>>>>
> >>>>>> When I execute this use case under load, I see that after ~2hours my
> >>>>>> requests fail and I get a series of errors like request
> >>>>>> timeouts(5seconds), invalid window update frame, and connection
> close
> >>>>>> exception on my client.
> >>>>>> On further debugging, I found that it's a Tomcat memory problem and
> it
> >>>>>> goes OOM after sometime under heavy load with multiple connections
> >> being
> >>>>>> re-established by the clients.
> >>>>>>
> >>>>>> image.png
> >>>>>>
> >>>>>> image.png
> >>>>>>
> >>>>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>>>
> >>>>> Embedded images get dropped by the list software. Post those images
> >>>>> somewhere we can see them.
> >>>>>
> >>>>>> Please let me know if you any experience with such a situation.
> Thanks
> >>>>>> in advance.
> >>>>>
> >>>>> Nothing comes to mind.
> >>>>>
> >>>>> I'll try some simple tests with HTTP/2.
> >>>>
> >>>> I don't see a memory leak (the memory is reclaimed eventually) but I
> do
> >>>> see possibilities to release memory associated with request processing
> >>>> sooner.
> >>>>
> >>>> Right now you need to allocate more memory to the Java process to
> enable
> >>>> Tomcat to handle the HTTP/2 load it is presented with.
> >>>>
> >>>> It looks like a reasonable chunk of memory is released when the
> >>>> Connection closes that could be released earlier when the associated
> >>>> Stream closes. I'll take a look at what can be done in that area. In
> the
> >>>> meantime, reducing the number of Streams you allow on a Connection
> >>>> before it is closed should reduce overall memory usage.
> >>>>
> >>>> Mark
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
Thanks.
I've looked at the code and I have tried various tests but I am unable
to re-create a memory leak.
The code used to (before I made a few changes this afternoon) retain a
lot more memory per Stream and it is possible that what you are seeing
is a system that doesn't have enough memory to achieve steady state.
If you are able to build the latest 9.0.x and test that, that could be
helpful. Alternatively, I could provide a test build for you to
experiment with.
Some additional questions that might aid understanding:
- What is the typical response size for one of these requests?
- How long does a typical test take to process?
- What are the GC roots for those RequestInfo objects?
Thanks again,
Mark
On 25/06/2020 15:10, Chirag Dewan wrote:
> Hi Mark,
>
> Its the default APR connector with 150 Threads.
>
> Chirag
>
> On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
>
>> On 25/06/2020 11:00, Chirag Dewan wrote:
>>> Thanks for the quick check Mark.
>>>
>>> These are the images I tried referring to:
>>>
>>> https://ibb.co/LzKtRgh
>>>
>>> https://ibb.co/2s7hqRL
>>>
>>> https://ibb.co/KmKj590
>>>
>>>
>>> The last one is the MAT screenshot showing many RequestInfo objects.
>>
>> Thanks. That certainly looks like a memory leak. I'll take a closer
>> look. Out of interest, how many threads is the Connector configured to use?
>>
>> Mark
>>
>>
>>>
>>>
>>> Thanks,
>>>
>>> Chirag
>>>
>>> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
>>>
>>>> On 24/06/2020 12:17, Mark Thomas wrote:
>>>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Update: We found that Tomcat goes OOM when a client closes and opens
>> new
>>>>>> connections every second. In the memory dump, we see a lot of
>>>>>> RequestInfo objects that are causing the memory spike.
>>>>>>
>>>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>>>> request timed out on my client). This seems like a bug to me.
>>>>>>
>>>>>> For better understanding, let me explain my use case again:
>>>>>>
>>>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>>>> requirement is to close a connection after a configurable(say 5000)
>>>>>> number of requests/streams and open a new connection that continues to
>>>>>> send requests. I close a connection by sending a GoAway frame.
>>>>>>
>>>>>> When I execute this use case under load, I see that after ~2hours my
>>>>>> requests fail and I get a series of errors like request
>>>>>> timeouts(5seconds), invalid window update frame, and connection close
>>>>>> exception on my client.
>>>>>> On further debugging, I found that it's a Tomcat memory problem and it
>>>>>> goes OOM after sometime under heavy load with multiple connections
>> being
>>>>>> re-established by the clients.
>>>>>>
>>>>>> image.png
>>>>>>
>>>>>> image.png
>>>>>>
>>>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>>>
>>>>> Embedded images get dropped by the list software. Post those images
>>>>> somewhere we can see them.
>>>>>
>>>>>> Please let me know if you any experience with such a situation. Thanks
>>>>>> in advance.
>>>>>
>>>>> Nothing comes to mind.
>>>>>
>>>>> I'll try some simple tests with HTTP/2.
>>>>
>>>> I don't see a memory leak (the memory is reclaimed eventually) but I do
>>>> see possibilities to release memory associated with request processing
>>>> sooner.
>>>>
>>>> Right now you need to allocate more memory to the Java process to enable
>>>> Tomcat to handle the HTTP/2 load it is presented with.
>>>>
>>>> It looks like a reasonable chunk of memory is released when the
>>>> Connection closes that could be released earlier when the associated
>>>> Stream closes. I'll take a look at what can be done in that area. In the
>>>> meantime, reducing the number of Streams you allow on a Connection
>>>> before it is closed should reduce overall memory usage.
>>>>
>>>> Mark
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Hi Mark,
Its the default APR connector with 150 Threads.
Chirag
On Thu, 25 Jun, 2020, 7:30 pm Mark Thomas, <ma...@apache.org> wrote:
> On 25/06/2020 11:00, Chirag Dewan wrote:
> > Thanks for the quick check Mark.
> >
> > These are the images I tried referring to:
> >
> > https://ibb.co/LzKtRgh
> >
> > https://ibb.co/2s7hqRL
> >
> > https://ibb.co/KmKj590
> >
> >
> > The last one is the MAT screenshot showing many RequestInfo objects.
>
> Thanks. That certainly looks like a memory leak. I'll take a closer
> look. Out of interest, how many threads is the Connector configured to use?
>
> Mark
>
>
> >
> >
> > Thanks,
> >
> > Chirag
> >
> > On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
> >
> >> On 24/06/2020 12:17, Mark Thomas wrote:
> >>> On 22/06/2020 11:06, Chirag Dewan wrote:
> >>>> Hi,
> >>>>
> >>>> Update: We found that Tomcat goes OOM when a client closes and opens
> new
> >>>> connections every second. In the memory dump, we see a lot of
> >>>> RequestInfo objects that are causing the memory spike.
> >>>>
> >>>> After a while, Tomcat goes OOM and start rejecting request(I get a
> >>>> request timed out on my client). This seems like a bug to me.
> >>>>
> >>>> For better understanding, let me explain my use case again:
> >>>>
> >>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >>>> requirement is to close a connection after a configurable(say 5000)
> >>>> number of requests/streams and open a new connection that continues to
> >>>> send requests. I close a connection by sending a GoAway frame.
> >>>>
> >>>> When I execute this use case under load, I see that after ~2hours my
> >>>> requests fail and I get a series of errors like request
> >>>> timeouts(5seconds), invalid window update frame, and connection close
> >>>> exception on my client.
> >>>> On further debugging, I found that it's a Tomcat memory problem and it
> >>>> goes OOM after sometime under heavy load with multiple connections
> being
> >>>> re-established by the clients.
> >>>>
> >>>> image.png
> >>>>
> >>>> image.png
> >>>>
> >>>> Is this a known issue? Or a known behavior with Tomcat?
> >>>
> >>> Embedded images get dropped by the list software. Post those images
> >>> somewhere we can see them.
> >>>
> >>>> Please let me know if you any experience with such a situation. Thanks
> >>>> in advance.
> >>>
> >>> Nothing comes to mind.
> >>>
> >>> I'll try some simple tests with HTTP/2.
> >>
> >> I don't see a memory leak (the memory is reclaimed eventually) but I do
> >> see possibilities to release memory associated with request processing
> >> sooner.
> >>
> >> Right now you need to allocate more memory to the Java process to enable
> >> Tomcat to handle the HTTP/2 load it is presented with.
> >>
> >> It looks like a reasonable chunk of memory is released when the
> >> Connection closes that could be released earlier when the associated
> >> Stream closes. I'll take a look at what can be done in that area. In the
> >> meantime, reducing the number of Streams you allow on a Connection
> >> before it is closed should reduce overall memory usage.
> >>
> >> Mark
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
On 25/06/2020 11:00, Chirag Dewan wrote:
> Thanks for the quick check Mark.
>
> These are the images I tried referring to:
>
> https://ibb.co/LzKtRgh
>
> https://ibb.co/2s7hqRL
>
> https://ibb.co/KmKj590
>
>
> The last one is the MAT screenshot showing many RequestInfo objects.
Thanks. That certainly looks like a memory leak. I'll take a closer
look. Out of interest, how many threads is the Connector configured to use?
Mark
>
>
> Thanks,
>
> Chirag
>
> On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
>
>> On 24/06/2020 12:17, Mark Thomas wrote:
>>> On 22/06/2020 11:06, Chirag Dewan wrote:
>>>> Hi,
>>>>
>>>> Update: We found that Tomcat goes OOM when a client closes and opens new
>>>> connections every second. In the memory dump, we see a lot of
>>>> RequestInfo objects that are causing the memory spike.
>>>>
>>>> After a while, Tomcat goes OOM and start rejecting request(I get a
>>>> request timed out on my client). This seems like a bug to me.
>>>>
>>>> For better understanding, let me explain my use case again:
>>>>
>>>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>>>> requirement is to close a connection after a configurable(say 5000)
>>>> number of requests/streams and open a new connection that continues to
>>>> send requests. I close a connection by sending a GoAway frame.
>>>>
>>>> When I execute this use case under load, I see that after ~2hours my
>>>> requests fail and I get a series of errors like request
>>>> timeouts(5seconds), invalid window update frame, and connection close
>>>> exception on my client.
>>>> On further debugging, I found that it's a Tomcat memory problem and it
>>>> goes OOM after sometime under heavy load with multiple connections being
>>>> re-established by the clients.
>>>>
>>>> image.png
>>>>
>>>> image.png
>>>>
>>>> Is this a known issue? Or a known behavior with Tomcat?
>>>
>>> Embedded images get dropped by the list software. Post those images
>>> somewhere we can see them.
>>>
>>>> Please let me know if you any experience with such a situation. Thanks
>>>> in advance.
>>>
>>> Nothing comes to mind.
>>>
>>> I'll try some simple tests with HTTP/2.
>>
>> I don't see a memory leak (the memory is reclaimed eventually) but I do
>> see possibilities to release memory associated with request processing
>> sooner.
>>
>> Right now you need to allocate more memory to the Java process to enable
>> Tomcat to handle the HTTP/2 load it is presented with.
>>
>> It looks like a reasonable chunk of memory is released when the
>> Connection closes that could be released earlier when the associated
>> Stream closes. I'll take a look at what can be done in that area. In the
>> meantime, reducing the number of Streams you allow on a Connection
>> before it is closed should reduce overall memory usage.
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Thanks for the quick check Mark.
These are the images I tried referring to:
https://ibb.co/LzKtRgh
https://ibb.co/2s7hqRL
https://ibb.co/KmKj590
The last one is the MAT screenshot showing many RequestInfo objects.
Thanks,
Chirag
On Wed, Jun 24, 2020 at 8:30 PM Mark Thomas <ma...@apache.org> wrote:
> On 24/06/2020 12:17, Mark Thomas wrote:
> > On 22/06/2020 11:06, Chirag Dewan wrote:
> >> Hi,
> >>
> >> Update: We found that Tomcat goes OOM when a client closes and opens new
> >> connections every second. In the memory dump, we see a lot of
> >> RequestInfo objects that are causing the memory spike.
> >>
> >> After a while, Tomcat goes OOM and start rejecting request(I get a
> >> request timed out on my client). This seems like a bug to me.
> >>
> >> For better understanding, let me explain my use case again:
> >>
> >> I have a jetty client that sends HTTP2 requests to Tomcat. My
> >> requirement is to close a connection after a configurable(say 5000)
> >> number of requests/streams and open a new connection that continues to
> >> send requests. I close a connection by sending a GoAway frame.
> >>
> >> When I execute this use case under load, I see that after ~2hours my
> >> requests fail and I get a series of errors like request
> >> timeouts(5seconds), invalid window update frame, and connection close
> >> exception on my client.
> >> On further debugging, I found that it's a Tomcat memory problem and it
> >> goes OOM after sometime under heavy load with multiple connections being
> >> re-established by the clients.
> >>
> >> image.png
> >>
> >> image.png
> >>
> >> Is this a known issue? Or a known behavior with Tomcat?
> >
> > Embedded images get dropped by the list software. Post those images
> > somewhere we can see them.
> >
> >> Please let me know if you any experience with such a situation. Thanks
> >> in advance.
> >
> > Nothing comes to mind.
> >
> > I'll try some simple tests with HTTP/2.
>
> I don't see a memory leak (the memory is reclaimed eventually) but I do
> see possibilities to release memory associated with request processing
> sooner.
>
> Right now you need to allocate more memory to the Java process to enable
> Tomcat to handle the HTTP/2 load it is presented with.
>
> It looks like a reasonable chunk of memory is released when the
> Connection closes that could be released earlier when the associated
> Stream closes. I'll take a look at what can be done in that area. In the
> meantime, reducing the number of Streams you allow on a Connection
> before it is closed should reduce overall memory usage.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
On 24/06/2020 12:17, Mark Thomas wrote:
> On 22/06/2020 11:06, Chirag Dewan wrote:
>> Hi,
>>
>> Update: We found that Tomcat goes OOM when a client closes and opens new
>> connections every second. In the memory dump, we see a lot of
>> RequestInfo objects that are causing the memory spike.
>>
>> After a while, Tomcat goes OOM and start rejecting request(I get a
>> request timed out on my client). This seems like a bug to me.
>>
>> For better understanding, let me explain my use case again:
>>
>> I have a jetty client that sends HTTP2 requests to Tomcat. My
>> requirement is to close a connection after a configurable(say 5000)
>> number of requests/streams and open a new connection that continues to
>> send requests. I close a connection by sending a GoAway frame.
>>
>> When I execute this use case under load, I see that after ~2hours my
>> requests fail and I get a series of errors like request
>> timeouts(5seconds), invalid window update frame, and connection close
>> exception on my client.
>> On further debugging, I found that it's a Tomcat memory problem and it
>> goes OOM after sometime under heavy load with multiple connections being
>> re-established by the clients.
>>
>> image.png
>>
>> image.png
>>
>> Is this a known issue? Or a known behavior with Tomcat?
>
> Embedded images get dropped by the list software. Post those images
> somewhere we can see them.
>
>> Please let me know if you any experience with such a situation. Thanks
>> in advance.
>
> Nothing comes to mind.
>
> I'll try some simple tests with HTTP/2.
I don't see a memory leak (the memory is reclaimed eventually) but I do
see possibilities to release memory associated with request processing
sooner.
Right now you need to allocate more memory to the Java process to enable
Tomcat to handle the HTTP/2 load it is presented with.
It looks like a reasonable chunk of memory is released when the
Connection closes that could be released earlier when the associated
Stream closes. I'll take a look at what can be done in that area. In the
meantime, reducing the number of Streams you allow on a Connection
before it is closed should reduce overall memory usage.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Mark Thomas <ma...@apache.org>.
On 22/06/2020 11:06, Chirag Dewan wrote:
> Hi,
>
> Update: We found that Tomcat goes OOM when a client closes and opens new
> connections every second. In the memory dump, we see a lot of
> RequestInfo objects that are causing the memory spike.
>
> After a while, Tomcat goes OOM and start rejecting request(I get a
> request timed out on my client). This seems like a bug to me.
>
> For better understanding, let me explain my use case again:
>
> I have a jetty client that sends HTTP2 requests to Tomcat. My
> requirement is to close a connection after a configurable(say 5000)
> number of requests/streams and open a new connection that continues to
> send requests. I close a connection by sending a GoAway frame.
>
> When I execute this use case under load, I see that after ~2hours my
> requests fail and I get a series of errors like request
> timeouts(5seconds), invalid window update frame, and connection close
> exception on my client.
> On further debugging, I found that it's a Tomcat memory problem and it
> goes OOM after sometime under heavy load with multiple connections being
> re-established by the clients.
>
> image.png
>
> image.png
>
> Is this a known issue? Or a known behavior with Tomcat?
Embedded images get dropped by the list software. Post those images
somewhere we can see them.
> Please let me know if you any experience with such a situation. Thanks
> in advance.
Nothing comes to mind.
I'll try some simple tests with HTTP/2.
Mark
>
> On Sun, Jun 14, 2020 at 11:30 AM Chirag Dewan <chirag.dewan22@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi,
>
> This is without load balancer actually. I am directly sending to Tomcat.
>
> Update:
>
> A part issue I found was to be 9.0.29. I observed that when request
> were timed out on client (2seconds), the client would send a RST
> frame. And the GoAway from Tomcat was perhaps a bug. In 9.0.36, RST
> frame is replied with a RST from Tomcat.
>
> Now the next part to troubleshoot is why after about an hour or so,
> requests are timed out at Tomcat.
>
> Could close to 100 HTTP2 connections per second cause this on Tomcat?
>
> Thanks
>
> On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, <michaelo@apache.org
> <ma...@apache.org>> wrote:
>
> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
> > Hi,
> >
> > We are observing that under high load, my clients start
> receiving a GoAway
> > frame with error:
> >
> > *Connection[{id}], Stream[{id}] an error occurred during
> processing that
> > was fatal to the connection.*
> >
> > Background : We have implemented our clients to close
> connections after
> > every 500-1000 requests (streams). This is a load balancer
> requirement that
> > we are working on and hence such a behavior. So with a
> throughput of around
> > 19k, almost 40 connections are closed and recreated every second.
> >
> > After we receive this frame, my clients start behaving
> erroneously. Before
> > this as well, my clients start sending RST_STREAM with
> canceled for each
> > request. Could this be due to the number of connections we
> open? Is it
> > related to the version of Tomcat? Or maybe my clients are
> misbehaving?
> >
> > Now since I only receive this under heavy load, I can't quite
> picture
> > enough reasons for this to happen.
> >
> > Any possible clues on where I should start looking?
> >
> > My Stack:
> > Server - Tomcat 9.0.29
> > Client - Jetty 9.x
>
> Does the same happen w/o the load balancer?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> <ma...@tomcat.apache.org>
> For additional commands, e-mail: users-help@tomcat.apache.org
> <ma...@tomcat.apache.org>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Hi,
Update: We found that Tomcat goes OOM when a client closes and opens new
connections every second. In the memory dump, we see a lot of RequestInfo
objects that are causing the memory spike.
After a while, Tomcat goes OOM and start rejecting request(I get a
request timed out on my client). This seems like a bug to me.
For better understanding, let me explain my use case again:
I have a jetty client that sends HTTP2 requests to Tomcat. My requirement
is to close a connection after a configurable(say 5000) number of
requests/streams and open a new connection that continues to send requests.
I close a connection by sending a GoAway frame.
When I execute this use case under load, I see that after ~2hours my
requests fail and I get a series of errors like request timeouts(5seconds),
invalid window update frame, and connection close exception on my client.
On further debugging, I found that it's a Tomcat memory problem and it goes
OOM after sometime under heavy load with multiple connections being
re-established by the clients.
[image: image.png]
[image: image.png]
Is this a known issue? Or a known behavior with Tomcat?
Please let me know if you any experience with such a situation. Thanks in
advance.
On Sun, Jun 14, 2020 at 11:30 AM Chirag Dewan <ch...@gmail.com>
wrote:
> Hi,
>
> This is without load balancer actually. I am directly sending to Tomcat.
>
> Update:
>
> A part issue I found was to be 9.0.29. I observed that when request were
> timed out on client (2seconds), the client would send a RST frame. And the
> GoAway from Tomcat was perhaps a bug. In 9.0.36, RST frame is replied with
> a RST from Tomcat.
>
> Now the next part to troubleshoot is why after about an hour or so,
> requests are timed out at Tomcat.
>
> Could close to 100 HTTP2 connections per second cause this on Tomcat?
>
> Thanks
>
> On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, <mi...@apache.org>
> wrote:
>
>> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
>> > Hi,
>> >
>> > We are observing that under high load, my clients start receiving a
>> GoAway
>> > frame with error:
>> >
>> > *Connection[{id}], Stream[{id}] an error occurred during processing that
>> > was fatal to the connection.*
>> >
>> > Background : We have implemented our clients to close connections after
>> > every 500-1000 requests (streams). This is a load balancer requirement
>> that
>> > we are working on and hence such a behavior. So with a throughput of
>> around
>> > 19k, almost 40 connections are closed and recreated every second.
>> >
>> > After we receive this frame, my clients start behaving erroneously.
>> Before
>> > this as well, my clients start sending RST_STREAM with canceled for each
>> > request. Could this be due to the number of connections we open? Is it
>> > related to the version of Tomcat? Or maybe my clients are misbehaving?
>> >
>> > Now since I only receive this under heavy load, I can't quite picture
>> > enough reasons for this to happen.
>> >
>> > Any possible clues on where I should start looking?
>> >
>> > My Stack:
>> > Server - Tomcat 9.0.29
>> > Client - Jetty 9.x
>>
>> Does the same happen w/o the load balancer?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Chirag Dewan <ch...@gmail.com>.
Hi,
This is without load balancer actually. I am directly sending to Tomcat.
Update:
A part issue I found was to be 9.0.29. I observed that when request were
timed out on client (2seconds), the client would send a RST frame. And the
GoAway from Tomcat was perhaps a bug. In 9.0.36, RST frame is replied with
a RST from Tomcat.
Now the next part to troubleshoot is why after about an hour or so,
requests are timed out at Tomcat.
Could close to 100 HTTP2 connections per second cause this on Tomcat?
Thanks
On Sun, 14 Jun, 2020, 12:27 AM Michael Osipov, <mi...@apache.org> wrote:
> Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
> > Hi,
> >
> > We are observing that under high load, my clients start receiving a
> GoAway
> > frame with error:
> >
> > *Connection[{id}], Stream[{id}] an error occurred during processing that
> > was fatal to the connection.*
> >
> > Background : We have implemented our clients to close connections after
> > every 500-1000 requests (streams). This is a load balancer requirement
> that
> > we are working on and hence such a behavior. So with a throughput of
> around
> > 19k, almost 40 connections are closed and recreated every second.
> >
> > After we receive this frame, my clients start behaving erroneously.
> Before
> > this as well, my clients start sending RST_STREAM with canceled for each
> > request. Could this be due to the number of connections we open? Is it
> > related to the version of Tomcat? Or maybe my clients are misbehaving?
> >
> > Now since I only receive this under heavy load, I can't quite picture
> > enough reasons for this to happen.
> >
> > Any possible clues on where I should start looking?
> >
> > My Stack:
> > Server - Tomcat 9.0.29
> > Client - Jetty 9.x
>
> Does the same happen w/o the load balancer?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Connection Closure due to Fatal Stream with HTTP2
Posted by Michael Osipov <mi...@apache.org>.
Am 2020-06-13 um 08:42 schrieb Chirag Dewan:
> Hi,
>
> We are observing that under high load, my clients start receiving a GoAway
> frame with error:
>
> *Connection[{id}], Stream[{id}] an error occurred during processing that
> was fatal to the connection.*
>
> Background : We have implemented our clients to close connections after
> every 500-1000 requests (streams). This is a load balancer requirement that
> we are working on and hence such a behavior. So with a throughput of around
> 19k, almost 40 connections are closed and recreated every second.
>
> After we receive this frame, my clients start behaving erroneously. Before
> this as well, my clients start sending RST_STREAM with canceled for each
> request. Could this be due to the number of connections we open? Is it
> related to the version of Tomcat? Or maybe my clients are misbehaving?
>
> Now since I only receive this under heavy load, I can't quite picture
> enough reasons for this to happen.
>
> Any possible clues on where I should start looking?
>
> My Stack:
> Server - Tomcat 9.0.29
> Client - Jetty 9.x
Does the same happen w/o the load balancer?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org