You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tapestry.apache.org by JumpStart <ge...@gmail.com> on 2019/05/16 01:33:17 UTC
Laggy response after a quiet time
Hi all,
My app is working brilliantly under load, but after a quiet time it can be very slow to respond, leading our first user of the day to tap the same thing multiple times, and the next thing you know is CPU hits 100% and is stuck there, and none of those requests returns a response. Nor do any new requests return a response. Apache logs show that all the requests time out after 60 secs, unanswered, and the health checkers start messaging the support staff.
Has anyone else experienced this kind of thing?
Perhaps it’s something to do with our infrastructure? We’re running Tapestry from an EAR in Wildfly in Docker in an AWS EC2 instance. Also in that EC2 instance is Apache HTTPD in Docker.
Any thoughts, please! It’s a crazy problem.
Cheers,
Geoff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Laggy response after a quiet time
Posted by JumpStart <ge...@gmail.com>.
Thanks Cezary. As I feared, there are many potential culprits to sniff out.
Geoff
> On 16 May 2019, at 10:05 pm, Cezary Biernacki <ce...@gmail.com> wrote:
>
> Obviously it would help to learn what CPU is doing. Does the application
> become unstuck and start working normally after initial spike of activity?
>
> I have not experienced exactly the same behaviour, and I have relatively
> similar technology - a Tapestry application running inside Docker on AWS
> EC2, thought it is a standalone process with Dropwizard/Jetty. But years
> ago I had related problems with a different WAR-based application that
> crashed after not being used some time. It was caused by the "tmpwatch"
> daemon on Linux deleting files in /tmp that had not been accessed after
> some time.
>
> I recommend checking if anything in your application can be described as
> "it can be removed if not used after predetermined amount of time" or "it
> needs refreshing from time to time and this determined on access".
> Potential culprits:
>
> - in-application caches, e.g. one built with Guava's Cache;
> - database connections or other TCP connections where an external
> service can terminate idle sessions;
> - an auto-scalling mechanism;
> - the application server determining that last day user sessions are
> expired and killing them with a listener doing something expensive on
> session's termination.
>
> Another option is weird garbage collection interactions, e.g. caused by
> non-trivial work in the "finalize" method somewhere in the application.
>
> Cezary
>
>
> On Thu, May 16, 2019 at 3:33 AM JumpStart <
> geoff.callender.jumpstart@gmail.com> wrote:
>
>> Hi all,
>>
>> My app is working brilliantly under load, but after a quiet time it can be
>> very slow to respond, leading our first user of the day to tap the same
>> thing multiple times, and the next thing you know is CPU hits 100% and is
>> stuck there, and none of those requests returns a response. Nor do any new
>> requests return a response. Apache logs show that all the requests time out
>> after 60 secs, unanswered, and the health checkers start messaging the
>> support staff.
>>
>> Has anyone else experienced this kind of thing?
>>
>> Perhaps it’s something to do with our infrastructure? We’re running
>> Tapestry from an EAR in Wildfly in Docker in an AWS EC2 instance. Also in
>> that EC2 instance is Apache HTTPD in Docker.
>>
>> Any thoughts, please! It’s a crazy problem.
>>
>> Cheers,
>>
>> Geoff
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
>> For additional commands, e-mail: users-help@tapestry.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Laggy response after a quiet time
Posted by Cezary Biernacki <ce...@gmail.com>.
Obviously it would help to learn what CPU is doing. Does the application
become unstuck and start working normally after initial spike of activity?
I have not experienced exactly the same behaviour, and I have relatively
similar technology - a Tapestry application running inside Docker on AWS
EC2, thought it is a standalone process with Dropwizard/Jetty. But years
ago I had related problems with a different WAR-based application that
crashed after not being used some time. It was caused by the "tmpwatch"
daemon on Linux deleting files in /tmp that had not been accessed after
some time.
I recommend checking if anything in your application can be described as
"it can be removed if not used after predetermined amount of time" or "it
needs refreshing from time to time and this determined on access".
Potential culprits:
- in-application caches, e.g. one built with Guava's Cache;
- database connections or other TCP connections where an external
service can terminate idle sessions;
- an auto-scalling mechanism;
- the application server determining that last day user sessions are
expired and killing them with a listener doing something expensive on
session's termination.
Another option is weird garbage collection interactions, e.g. caused by
non-trivial work in the "finalize" method somewhere in the application.
Cezary
On Thu, May 16, 2019 at 3:33 AM JumpStart <
geoff.callender.jumpstart@gmail.com> wrote:
> Hi all,
>
> My app is working brilliantly under load, but after a quiet time it can be
> very slow to respond, leading our first user of the day to tap the same
> thing multiple times, and the next thing you know is CPU hits 100% and is
> stuck there, and none of those requests returns a response. Nor do any new
> requests return a response. Apache logs show that all the requests time out
> after 60 secs, unanswered, and the health checkers start messaging the
> support staff.
>
> Has anyone else experienced this kind of thing?
>
> Perhaps it’s something to do with our infrastructure? We’re running
> Tapestry from an EAR in Wildfly in Docker in an AWS EC2 instance. Also in
> that EC2 instance is Apache HTTPD in Docker.
>
> Any thoughts, please! It’s a crazy problem.
>
> Cheers,
>
> Geoff
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
>
Re: Laggy response after a quiet time
Posted by JumpStart <ge...@gmail.com>.
Thanks guys. These are all helpful suggestions that I will look into.
Geoff
> On 16 May 2019, at 8:59 pm, Dakszewicz Diego <tn...@gmail.com> wrote:
>
> Hi Dmitry!
> Tapestry since 5.3 minify css and js on a fly. That's why your cpu hits
> 100%, at startup when tapestry warm up resources on first request and when
> your site has low traffic.
> A workaround to solve this problem is disable this feature.
> Take a look at your AppModule in
>
> @Contribute(ResourceMinimizer.class)
> public static void
> contributeMinimizers(MappedConfiguration<String, ResourceMinimizer>
> configuration) {}
>
> You can implement your own minimizer or just disable it. I prefer minify my
> resources at deploy time. It's recommended if you have to warm up a lot of
> apps hosted on the same server.
> Cheers,
> Diego
>
> El jue., 16 may. 2019 a las 4:59, Dmitry Gusev (<dm...@gmail.com>)
> escribió:
>
>> Hi,
>>
>> I'd also try to collect thread dump when you see this happening to get an
>> idea where the CPU is wasted.
>>
>> On Thu, May 16, 2019 at 10:38 AM Mats Andersson <mats.andersson@ronsoft.se
>>>
>> wrote:
>>
>>> Do you have some scheduled services that could eat memory? I would
>>> suspect that the garbage collector is using most of that CPU. Monitor
>>> the GC and how memory is used over time. Maybe there is a mismatch
>>> between the memory configurations in Docker and the JVM.
>>>
>>> It seems like it is repeatable, that is good for troubleshooting at
>> least.
>>>
>>> Mats
>>>
>>>
>>> On 2019-05-16 03:33, JumpStart wrote:
>>>> Hi all,
>>>>
>>>> My app is working brilliantly under load, but after a quiet time it can
>>> be very slow to respond, leading our first user of the day to tap the
>> same
>>> thing multiple times, and the next thing you know is CPU hits 100% and is
>>> stuck there, and none of those requests returns a response. Nor do any
>> new
>>> requests return a response. Apache logs show that all the requests time
>> out
>>> after 60 secs, unanswered, and the health checkers start messaging the
>>> support staff.
>>>>
>>>> Has anyone else experienced this kind of thing?
>>>>
>>>> Perhaps it’s something to do with our infrastructure? We’re running
>>> Tapestry from an EAR in Wildfly in Docker in an AWS EC2 instance. Also in
>>> that EC2 instance is Apache HTTPD in Docker.
>>>>
>>>> Any thoughts, please! It’s a crazy problem.
>>>>
>>>> Cheers,
>>>>
>>>> Geoff
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
>>>> For additional commands, e-mail: users-help@tapestry.apache.org
>>>>
>>> --
>>> ---------------------- Mats Andersson | Ronsoft AB | +46(0)73 368 79 82
>>>
>>>
>>
>> --
>> Dmitry Gusev
>>
>> AnjLab Team
>> http://anjlab.com
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
For additional commands, e-mail: users-help@tapestry.apache.org
Re: Laggy response after a quiet time
Posted by Dakszewicz Diego <tn...@gmail.com>.
Hi Dmitry!
Tapestry since 5.3 minify css and js on a fly. That's why your cpu hits
100%, at startup when tapestry warm up resources on first request and when
your site has low traffic.
A workaround to solve this problem is disable this feature.
Take a look at your AppModule in
@Contribute(ResourceMinimizer.class)
public static void
contributeMinimizers(MappedConfiguration<String, ResourceMinimizer>
configuration) {}
You can implement your own minimizer or just disable it. I prefer minify my
resources at deploy time. It's recommended if you have to warm up a lot of
apps hosted on the same server.
Cheers,
Diego
El jue., 16 may. 2019 a las 4:59, Dmitry Gusev (<dm...@gmail.com>)
escribió:
> Hi,
>
> I'd also try to collect thread dump when you see this happening to get an
> idea where the CPU is wasted.
>
> On Thu, May 16, 2019 at 10:38 AM Mats Andersson <mats.andersson@ronsoft.se
> >
> wrote:
>
> > Do you have some scheduled services that could eat memory? I would
> > suspect that the garbage collector is using most of that CPU. Monitor
> > the GC and how memory is used over time. Maybe there is a mismatch
> > between the memory configurations in Docker and the JVM.
> >
> > It seems like it is repeatable, that is good for troubleshooting at
> least.
> >
> > Mats
> >
> >
> > On 2019-05-16 03:33, JumpStart wrote:
> > > Hi all,
> > >
> > > My app is working brilliantly under load, but after a quiet time it can
> > be very slow to respond, leading our first user of the day to tap the
> same
> > thing multiple times, and the next thing you know is CPU hits 100% and is
> > stuck there, and none of those requests returns a response. Nor do any
> new
> > requests return a response. Apache logs show that all the requests time
> out
> > after 60 secs, unanswered, and the health checkers start messaging the
> > support staff.
> > >
> > > Has anyone else experienced this kind of thing?
> > >
> > > Perhaps it’s something to do with our infrastructure? We’re running
> > Tapestry from an EAR in Wildfly in Docker in an AWS EC2 instance. Also in
> > that EC2 instance is Apache HTTPD in Docker.
> > >
> > > Any thoughts, please! It’s a crazy problem.
> > >
> > > Cheers,
> > >
> > > Geoff
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> > > For additional commands, e-mail: users-help@tapestry.apache.org
> > >
> > --
> > ---------------------- Mats Andersson | Ronsoft AB | +46(0)73 368 79 82
> >
> >
>
> --
> Dmitry Gusev
>
> AnjLab Team
> http://anjlab.com
>
Re: Laggy response after a quiet time
Posted by Dmitry Gusev <dm...@gmail.com>.
Hi,
I'd also try to collect thread dump when you see this happening to get an
idea where the CPU is wasted.
On Thu, May 16, 2019 at 10:38 AM Mats Andersson <ma...@ronsoft.se>
wrote:
> Do you have some scheduled services that could eat memory? I would
> suspect that the garbage collector is using most of that CPU. Monitor
> the GC and how memory is used over time. Maybe there is a mismatch
> between the memory configurations in Docker and the JVM.
>
> It seems like it is repeatable, that is good for troubleshooting at least.
>
> Mats
>
>
> On 2019-05-16 03:33, JumpStart wrote:
> > Hi all,
> >
> > My app is working brilliantly under load, but after a quiet time it can
> be very slow to respond, leading our first user of the day to tap the same
> thing multiple times, and the next thing you know is CPU hits 100% and is
> stuck there, and none of those requests returns a response. Nor do any new
> requests return a response. Apache logs show that all the requests time out
> after 60 secs, unanswered, and the health checkers start messaging the
> support staff.
> >
> > Has anyone else experienced this kind of thing?
> >
> > Perhaps it’s something to do with our infrastructure? We’re running
> Tapestry from an EAR in Wildfly in Docker in an AWS EC2 instance. Also in
> that EC2 instance is Apache HTTPD in Docker.
> >
> > Any thoughts, please! It’s a crazy problem.
> >
> > Cheers,
> >
> > Geoff
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> > For additional commands, e-mail: users-help@tapestry.apache.org
> >
> --
> ---------------------- Mats Andersson | Ronsoft AB | +46(0)73 368 79 82
>
>
--
Dmitry Gusev
AnjLab Team
http://anjlab.com
Re: Laggy response after a quiet time
Posted by Mats Andersson <ma...@ronsoft.se>.
Do you have some scheduled services that could eat memory? I would
suspect that the garbage collector is using most of that CPU. Monitor
the GC and how memory is used over time. Maybe there is a mismatch
between the memory configurations in Docker and the JVM.
It seems like it is repeatable, that is good for troubleshooting at least.
Mats
On 2019-05-16 03:33, JumpStart wrote:
> Hi all,
>
> My app is working brilliantly under load, but after a quiet time it can be very slow to respond, leading our first user of the day to tap the same thing multiple times, and the next thing you know is CPU hits 100% and is stuck there, and none of those requests returns a response. Nor do any new requests return a response. Apache logs show that all the requests time out after 60 secs, unanswered, and the health checkers start messaging the support staff.
>
> Has anyone else experienced this kind of thing?
>
> Perhaps it’s something to do with our infrastructure? We’re running Tapestry from an EAR in Wildfly in Docker in an AWS EC2 instance. Also in that EC2 instance is Apache HTTPD in Docker.
>
> Any thoughts, please! It’s a crazy problem.
>
> Cheers,
>
> Geoff
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tapestry.apache.org
> For additional commands, e-mail: users-help@tapestry.apache.org
>
--
---------------------- Mats Andersson | Ronsoft AB | +46(0)73 368 79 82