You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ivano Luberti <lu...@archicoop.it.INVALID> on 2021/07/21 14:34:17 UTC
tomcat 8.5.57 stops killing sessions after some time
Hello, I'm new to the list but befeore writing this I have searched the
users mailing list without finding anything useful.
I have an instance of Apache Tomcat/8.5.57 running on a CentOS machine
with java 1.7.0_261
Several webapps run on this instance.
I have noticed that after a few days tomcat stops killing expired sessions.
Trying to understand where the issue is located , I have enabled debug
logging of org.apache.catalina.session package with:
java.util.logging.ConsoleHandler.level = FINEST
org.apache.catalina.session.level = FINEST
So after the latest restart i could observe in catalina.out many log
rows like
21-Jul-2021 10:02:00.702 FINE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.session.ManagerBase.processExpires Start expire
sessions StandardManager at 1626861720702 sessioncount 1
21-Jul-2021 10:02:00.702 FINE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.session.ManagerBase.processExpires End expire
sessions StandardManager processingTime 0 expired sessions: 0
Today, after 4 days without issues tomcat stopped again killing expired
sessions and at the same time no more logs of
org.apache.catalina.session.ManagerBase.processExpires
are reported.
There is no other related log event , simply the last row reported is
21-Jul-2021 12:38:39.370 FINE
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.session.ManagerBase.processExpires Start expire
sessions StandardManager at 1626871119370 sessioncount 7
Any clue on why org.apache.catalina.session.ManagerBase is not called
anymore?
Suggestion on how to diagnose?
Any chance that some exception occurring in
HttpSessionListener.sessionCreated
or
HttpSessionListener.sessionDestroyed
or
HttpSessionBindingListener.valueBound
or
HttpSessionBindingListener.valueUnbound
can terminate ManagerBase but without generating any logging?
Those are the only interaction with session handling I'm aware of.
I have the same webapps running on other tomcat instances, but on older
tomcat version and there I have no issue of ths kind.
TIA for any hint.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Ivano Luberti <lu...@archicoop.it.INVALID>.
Il 30/07/2021 14:54, Konstantin Kolinko ha scritto:
>
> Additional note: the "backgroundProcessorDelay" on a Context can be
> used to make a web application use its own background thread (instead
> of reusing the one inherited from its parent container). It won't
> help with fixing a deadlock in a particular web application, but it
> will isolate the failure so that other web applications on the same
> server won't suffer.
Good suggestion Kostantin, thanks
> https://tomcat.apache.org/tomcat-8.5-doc/config/context.html
>
> Best regards,
> Konstantin Kolinko
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Archimede Informatica tratta i dati personali in conformità a quanto
stabilito dal Regolamento UE n. 2016/679 (GDPR) e dal D. Lgs. 30 giugno
2003 n. 196
per come modificato dal D.Lgs. 10 agosto 2018 n. 101.
Informativa completa
<http://www.archicoop.it/fileadmin/pdf/InformativaTrattamentoDatiPersonali.pdf>
dott. Ivano Mario Luberti
Archimede Informatica società cooperativa a r. l.
Via Gereschi 36, 56127 Pisa
tel.: +39 050/580959 | fax: +39 050/8932061
web: www.archicoop.it
linkedin: www.linkedin.com/in/ivanoluberti
facebook: www.facebook.com/archimedeinformaticapisa/
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Konstantin Kolinko <kn...@gmail.com>.
> [....]
> Il 26/07/2021 23:15, Christopher Schultz ha scritto:
> > Ivano,
> >
> > On 7/23/21 02:20, Ivano Luberti wrote:
> >> I have found the issue: one of the webapps has several thread locked
> >> on a MultiThreadedHttpConnectionManager during sessionUnbound and
> >> unboundEvents.
> >>
> >> So the background process killing the sessions is stuck as well.
> >
> > This is a very easy problem to encounter. Session event-handlers
> > really need to be bulletproof and execute very quickly.
> >
> > If you need a long-running process to trigger, use an executor service
> > or something where you can capture the information to be used and
> > off-load it onto another thread. Remember that the session cannot be
> > used after the event-handler completes, so capture whatever
> > information you need from the session FIRST, then dispatch to another
> > thread, then return from the event-handler.
> >
> > -chris
> >
> >> Il 21/07/2021 18:13, Mark Thomas ha scritto:
> >>> On 21/07/2021 16:00, Ivano Luberti wrote:
> >>>>
> >>>> Il 21/07/2021 16:44, Mark Thomas ha scritto:
> >>>>> Take 3 thread dumps 5 seconds apart and post them here.
> >>>>
> >>>> How to take thread dumps?
> >>>>
> >>>> kill -3 <pid> ?
> >>>
> >>> That will work. There are lots of ways. This is most of them:
> >>>
> >>> https://www.baeldung.com/java-thread-dump
I usually refer to troubleshooting pages of Tomcat FAQ:
https://cwiki.apache.org/confluence/display/TOMCAT/Troubleshooting+and+Diagnostics#TroubleshootingandDiagnostics-Techniques&Reference
https://cwiki.apache.org/confluence/display/TOMCAT/HowTo#HowTo-HowdoIobtainathreaddumpofmyrunningwebapp?
Additional note: the "backgroundProcessorDelay" on a Context can be
used to make a web application use its own background thread (instead
of reusing the one inherited from its parent container). It won't
help with fixing a deadlock in a particular web application, but it
will isolate the failure so that other web applications on the same
server won't suffer.
https://tomcat.apache.org/tomcat-8.5-doc/config/context.html
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Ivano Luberti <lu...@archicoop.it.INVALID>.
Thanks Chris.
In this case is a bug in the (old version of the) Axis2 library we use
to build and run a web service client.
I have upgraded Axis2 and waiting to see if the issue resurfaces.
The idea to use a new thread to manage the session event handles doesn't
apply here cause of application constraints and because operations are
quite simple.
But thanks for the advice.
Cheers
Il 26/07/2021 23:15, Christopher Schultz ha scritto:
> Ivano,
>
> On 7/23/21 02:20, Ivano Luberti wrote:
>> I have found the issue: one of the webapps has several thread locked
>> on a MultiThreadedHttpConnectionManager during sessionUnbound and
>> unboundEvents.
>>
>> So the background process killing the sessions is stuck as well.
>
> This is a very easy problem to encounter. Session event-handlers
> really need to be bulletproof and execute very quickly.
>
> If you need a long-running process to trigger, use an executor service
> or something where you can capture the information to be used and
> off-load it onto another thread. Remember that the session cannot be
> used after the event-handler completes, so capture whatever
> information you need from the session FIRST, then dispatch to another
> thread, then return from the event-handler.
>
> -chris
>
>> Il 21/07/2021 18:13, Mark Thomas ha scritto:
>>> On 21/07/2021 16:00, Ivano Luberti wrote:
>>>>
>>>> Il 21/07/2021 16:44, Mark Thomas ha scritto:
>>>>> Take 3 thread dumps 5 seconds apart and post them here.
>>>>
>>>> How to take thread dumps?
>>>>
>>>> kill -3 <pid> ?
>>>
>>> That will work. There are lots of ways. This is most of them:
>>>
>>> https://www.baeldung.com/java-thread-dump
>>>
>>> Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Archimede Informatica tratta i dati personali in conformità a quanto
stabilito dal Regolamento UE n. 2016/679 (GDPR) e dal D. Lgs. 30 giugno
2003 n. 196
per come modificato dal D.Lgs. 10 agosto 2018 n. 101.
Informativa completa
<http://www.archicoop.it/fileadmin/pdf/InformativaTrattamentoDatiPersonali.pdf>
dott. Ivano Mario Luberti
Archimede Informatica società cooperativa a r. l.
Via Gereschi 36, 56127 Pisa
tel.: +39 050/580959 | fax: +39 050/8932061
web: www.archicoop.it
linkedin: www.linkedin.com/in/ivanoluberti
facebook: www.facebook.com/archimedeinformaticapisa/
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Ivano,
On 7/23/21 02:20, Ivano Luberti wrote:
> I have found the issue: one of the webapps has several thread locked on
> a MultiThreadedHttpConnectionManager during sessionUnbound and
> unboundEvents.
>
> So the background process killing the sessions is stuck as well.
This is a very easy problem to encounter. Session event-handlers really
need to be bulletproof and execute very quickly.
If you need a long-running process to trigger, use an executor service
or something where you can capture the information to be used and
off-load it onto another thread. Remember that the session cannot be
used after the event-handler completes, so capture whatever information
you need from the session FIRST, then dispatch to another thread, then
return from the event-handler.
-chris
> Il 21/07/2021 18:13, Mark Thomas ha scritto:
>> On 21/07/2021 16:00, Ivano Luberti wrote:
>>>
>>> Il 21/07/2021 16:44, Mark Thomas ha scritto:
>>>> Take 3 thread dumps 5 seconds apart and post them here.
>>>
>>> How to take thread dumps?
>>>
>>> kill -3 <pid> ?
>>
>> That will work. There are lots of ways. This is most of them:
>>
>> https://www.baeldung.com/java-thread-dump
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Ivano Luberti <lu...@archicoop.it.INVALID>.
I have found the issue: one of the webapps has several thread locked on
a MultiThreadedHttpConnectionManager during sessionUnbound and
unboundEvents.
So the background process killing the sessions is stuck as well.
Thanks for your support.
Il 21/07/2021 18:13, Mark Thomas ha scritto:
> On 21/07/2021 16:00, Ivano Luberti wrote:
>>
>> Il 21/07/2021 16:44, Mark Thomas ha scritto:
>>> Take 3 thread dumps 5 seconds apart and post them here.
>>
>> How to take thread dumps?
>>
>> kill -3 <pid> ?
>
> That will work. There are lots of ways. This is most of them:
>
> https://www.baeldung.com/java-thread-dump
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Archimede Informatica tratta i dati personali in conformità a quanto
stabilito dal Regolamento UE n. 2016/679 (GDPR) e dal D. Lgs. 30 giugno
2003 n. 196
per come modificato dal D.Lgs. 10 agosto 2018 n. 101.
Informativa completa
<http://www.archicoop.it/fileadmin/pdf/InformativaTrattamentoDatiPersonali.pdf>
dott. Ivano Mario Luberti
Archimede Informatica società cooperativa a r. l.
Via Gereschi 36, 56127 Pisa
tel.: +39 050/580959 | fax: +39 050/8932061
web: www.archicoop.it
linkedin: www.linkedin.com/in/ivanoluberti
facebook: www.facebook.com/archimedeinformaticapisa/
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Mark Thomas <ma...@apache.org>.
On 21/07/2021 16:00, Ivano Luberti wrote:
>
> Il 21/07/2021 16:44, Mark Thomas ha scritto:
>> Take 3 thread dumps 5 seconds apart and post them here.
>
> How to take thread dumps?
>
> kill -3 <pid> ?
That will work. There are lots of ways. This is most of them:
https://www.baeldung.com/java-thread-dump
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Ivano Luberti <lu...@archicoop.it.INVALID>.
Il 21/07/2021 16:44, Mark Thomas ha scritto:
> Take 3 thread dumps 5 seconds apart and post them here.
How to take thread dumps?
kill -3 <pid> ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 8.5.57 stops killing sessions after some time
Posted by Mark Thomas <ma...@apache.org>.
On 21/07/2021 15:34, Ivano Luberti wrote:
> Hello, I'm new to the list but befeore writing this I have searched the
> users mailing list without finding anything useful.
>
> I have an instance of Apache Tomcat/8.5.57 running on a CentOS machine
> with java 1.7.0_261
>
> Several webapps run on this instance.
>
> I have noticed that after a few days tomcat stops killing expired sessions.
>
> Trying to understand where the issue is located , I have enabled debug
> logging of org.apache.catalina.session package with:
>
> java.util.logging.ConsoleHandler.level = FINEST
>
> org.apache.catalina.session.level = FINEST
>
> So after the latest restart i could observe in catalina.out many log
> rows like
>
> 21-Jul-2021 10:02:00.702 FINE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.session.ManagerBase.processExpires Start expire
> sessions StandardManager at 1626861720702 sessioncount 1
> 21-Jul-2021 10:02:00.702 FINE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.session.ManagerBase.processExpires End expire
> sessions StandardManager processingTime 0 expired sessions: 0
>
> Today, after 4 days without issues tomcat stopped again killing expired
> sessions and at the same time no more logs of
>
> org.apache.catalina.session.ManagerBase.processExpires
>
> are reported.
>
> There is no other related log event , simply the last row reported is
>
> 21-Jul-2021 12:38:39.370 FINE
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.session.ManagerBase.processExpires Start expire
> sessions StandardManager at 1626871119370 sessioncount 7
>
> Any clue on why org.apache.catalina.session.ManagerBase is not called
> anymore?
Deadlock during background processing maybe?
> Suggestion on how to diagnose?
Take 3 thread dumps 5 seconds apart and post them here.
> Any chance that some exception occurring in
>
> HttpSessionListener.sessionCreated
> or
> HttpSessionListener.sessionDestroyed
> or
> HttpSessionBindingListener.valueBound
> or
> HttpSessionBindingListener.valueUnbound
>
> can terminate ManagerBase but without generating any logging?
Not that I can see. If the thread fails, it should be automatically
restarted. An OOME might be able to trigger this but that looks pretty
far fetched at this point.
> Those are the only interaction with session handling I'm aware of.
>
> I have the same webapps running on other tomcat instances, but on older
> tomcat version and there I have no issue of ths kind.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org