You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2014/06/05 17:19:21 UTC
Background thread died; no errors in log; invoking backgroundProcess
via JMX has no effect
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
One of dev instances this morning is screaming because I have hundreds
of active sessions. I checked, and it looks like the background
processor thread for the context is not running, therefore no sessions
are expiring.
Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
Here are the still-running threads:
"catalina-exec-946"
"Attach Listener"
"MySQL Statement Cancellation Timer"
"catalina-exec-338"
"catalina-exec-337"
"catalina-exec-5"
"catalina-exec-4"
"catalina-exec-3"
"http-nio-127.0.0.1-8217-Acceptor-0"
"http-nio-127.0.0.1-8217-ClientPoller-1"
"http-nio-127.0.0.1-8217-ClientPoller-0"
"ajp-bio-8215-AsyncTimeout"
"ajp-bio-8215-Acceptor-0"
"Memcached IO over {MemcachedConnection to localhost/127.0.0.1:11211}"
"Abandoned connection cleanup thread"
"Thread-3"
"NioBlockingSelector.BlockPoller-1"
"GC Daemon"
"Service Thread"
"C2 CompilerThread1"
"C2 CompilerThread0"
"Signal Dispatcher"
"Finalizer"
"Reference Handler"
"main"
"VM Thread"
"GC task thread#0 (ParallelGC)"
"GC task thread#1 (ParallelGC)"
"VM Periodic Task Thread"
I tried invoking "backgroundProcess" on my Manager bean via JMX but it
didn't seem to actually clean anything up. Is there any hope of
recovery without bouncing the web application?
Also, it would be nice to get a notification if the thread is dying
due to some exception. I searched-though catalina.out and found
nothing relevant. Any suggestions?
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTkIp2AAoJEBzwKT+lPKRYxOUQAJVMmWhZEimBNZiSwcgP2lNA
PET0guRv9x8b9A75UcLjPU1E69OCZ1AxpsT0W8eKLhzI8ehmv/COb6JDIIW79cuL
ksOSXeFwsPQfv/f0rB2byDmIrMVBg+053PLU3v3nWWp0VnLsdrr2oGfdvyjUmdtE
WuqhmGtVqzpOuZZixq3xi2sYWwmwATA1DdsTWllat6mdbgC2289S9lv8w7a/SXO0
f3aYW+AFxnX3nPN4ldbpbrUVOUfxKHXlMdjf4Rg1Z/aTIWNrInlJSAgiUjD6hY5P
rm/F83/zbsCfm7mRDPVjw425SQ3tpYX7K6p7bSoPkCwip9brURUyuIz4aSkLv27e
fO+rgm8P3/jdUmyfh6UT+mB5li2uBS6vPHu87lrIH1c5o65x3/+S0FgNA5Q5tJNC
GYti74FNwwga+g7cOMsItnZrOExm7xnpDifctGkMV3V7S2mG66iCgDVD7vk4B8F/
tBOSR+i1OP6RMwGLbnr2OW/QFKTnksN8jBlAFF4osXIPf1QrcUwLJA7o2z3ynqlR
ykHaolvZ2HU25gieAUfKy7ZRj7dJ61aAP3Ak7Rpr9Krno0GxlKTS53JeYg1mNGsp
4goBt8jdvyMMMA8ZcV7HfAUfPAbJn/2ehmkCLBqzMu0Un3OeRADvZZrI308wTPSG
q9r0JhsDD/BZUhE47Iwj
=Az1U
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Background thread died; no errors in log; invoking backgroundProcess
via JMX has no effect
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Konstantin,
On 6/5/14, 4:27 PM, Konstantin Kolinko wrote:
> 2014-06-05 23:06 GMT+04:00 Christopher Schultz
> <ch...@christopherschultz.net>:
>> On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
>>> 2014-06-05 19:19 GMT+04:00 Christopher Schultz
>>> <ch...@christopherschultz.net>:
>>>> I tried invoking "backgroundProcess" on my Manager bean via
>>>> JMX but it didn't seem to actually clean anything up. Is
>>>> there any hope of recovery without bouncing the web
>>>> application?
>>>>
>>>> Also, it would be nice to get a notification if the thread
>>>> is dying due to some exception. I searched-though
>>>> catalina.out and found nothing relevant. Any suggestions?
>>>
>>> In general, dying threads are handled via
>>> Thread.setUncaughtExceptionHandler(...) and
>>> ThreadGroup.uncaughtException(...)
>>>
>>> The default behaviour in JRE it so log to System.err. So
>>> catalina.out is the place where I expect it to be written.
>>
>> That's what I would have thought, too. I didn't see anything,
>> unfortunately.
>>
>>> (It is not nice as it bypasses logging framework, but that is
>>> how it is currently).
>>>
>>> A unusual culprit is OutOfMemoryError.
>>
>> I think you mean "a usual culprit" and I agree: this has happened
>> in the past where the background processor thread dies due to
>> OOME.
>>
>
> Yes, a typo. I meant "a usual one".
>
>>>> I tried invoking "backgroundProcess" on my Manager bean via
>>>> JMX but it didn't seem to actually clean anything up. Is
>>>> there any hope of recovery without bouncing the web
>>>> application?
>>>
>>> The background thread is started via threadStart() which is
>>> called from ContainerBase.startInternal() /
>>> StandardContext.startInternal().
>>>
>>> So I see no hope of recovering unless the container that owns
>>> the thread is stopped/started. I think that by default the
>>> thread is started by Engine, so Engine needs to be bounced
>>> here.
>>
>> The JMX method "backgroundProcess" should call
>> ManagerBase.backgroundProcess which should go ahead and expire
>> the sessions. I don't see a reason why it wouldn't work.
>> Unfortunately, Tomcat's Manager wasn't in debug-log mode or I
>> would have been able to see what happened when I invoked
>> backgroundProcess.
>>
>
> 1. If you have Manager webapp installed, it has API to expire
> sessions (on the "list" page in HTML gui)
>
> 2. I think logging level can be manipulated via JMX.
I hadn't thought of that. Next time. ;)
>>> BTW, javadoc for ContainerBase.backgroundProcess() says "This
>>> method will be invoked inside the classloading context of this
>>> container." I think that is not true when it is being invoked
>>> via JMX.
>>
>> This is ManagerBase.backgroundProcess, which does not have the
>> same documentation. While ContainerBase.backgroundProcess would
>> usually invoke ManagerBase.backgroundProcess, using JMX goes
>> directly to ManagerBase.
>
> OK.
>
>> I'm still curious why invoking ManagerBase.backgroundProcess via
>> JMX had no effect. I'm not sure what the "count" and
>> "processExpiresFrequency" values were at the time, so my calls
>> could have been rejected due to
>> background-processing-throttling.
>
> //ManagerBase.java @Override public void backgroundProcess() {
> count = (count + 1) % processExpiresFrequency; if (count == 0)
> processExpires(); }
>
> processExpiresFrequency should be readable via JMX.
>
> Default is '6', so you have to call the method 6 times in a row.
Aah. I gave up after 3. Next time.
Thanks for the info!
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTkOEWAAoJEBzwKT+lPKRY9hwP/Azf+FdBNpEl/V29SjuVfAb/
GTkEwzcVyxFh0uzr1wydcjdw8vE/SeVYcL0++dWmjzV6qGPeVLY0/wbrASBfJckc
Cka8gbLpSnIaVkrGqEAqWvfisSXYY2UGpCh2raOzgRW8/kqKcwriNrxRyZ7vPXXo
v4Ql+J4mjBC+XN4RRqHnz//l0bRBuwYGUgfeew7Tv5hO32PfGV2GS6FtgsbceIP7
GDRpqFoBo9xQtmgYHvHIVYrRD7zSBFcf72YZdY23PIlQum6TdvxWjkt0KipNrRE8
N7mDBprddiuRv1q6vNjWJbkz+y+YROd4PlHpK/sCxhdQMfvpl4SpNXp2Uo0b6cBB
KiFjJ6i9HUcEbaHrRZXCS+UoOxBOhQM/WBRtPXv/Vy+k7hIyjtfs1QCojrwnQQcl
EMff7iDdukk+m2FlnK3SdbxJALGtVh1Y+Q3eYa+X3IqBoz7/RQDt1tBUjpm4eKkF
G+cg7/+JubJJh+2TMOHzi7YFsBo978TMMO3rmu+Bl3E4YHayzt0RMfde2V0Dj4jD
WJCQOFz/JYHRjZoeY/lJSx0Y9kydfNeqF/aQ5hlF1+c2PKVYVkwp5NG5H48yXjpo
E3HcDV0nSV69VmbWQKiZpGn74B5J9tHTLQ7KYFJRH0BpyOk+QZc0Zzh2zTtAnbyq
tRA2omkbjAifP0PA0IMg
=jS15
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Background thread died; no errors in log; invoking
backgroundProcess via JMX has no effect
Posted by Konstantin Kolinko <kn...@gmail.com>.
2014-06-05 23:06 GMT+04:00 Christopher Schultz <ch...@christopherschultz.net>:
> On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
>> 2014-06-05 19:19 GMT+04:00 Christopher Schultz
>> <ch...@christopherschultz.net>:
>>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>>> but it didn't seem to actually clean anything up. Is there any
>>> hope of recovery without bouncing the web application?
>>>
>>> Also, it would be nice to get a notification if the thread is
>>> dying due to some exception. I searched-though catalina.out and
>>> found nothing relevant. Any suggestions?
>>
>> In general, dying threads are handled via
>> Thread.setUncaughtExceptionHandler(...) and
>> ThreadGroup.uncaughtException(...)
>>
>> The default behaviour in JRE it so log to System.err. So
>> catalina.out is the place where I expect it to be written.
>
> That's what I would have thought, too. I didn't see anything,
> unfortunately.
>
>> (It is not nice as it bypasses logging framework, but that is how
>> it is currently).
>>
>> A unusual culprit is OutOfMemoryError.
>
> I think you mean "a usual culprit" and I agree: this has happened in
> the past where the background processor thread dies due to OOME.
>
Yes, a typo. I meant "a usual one".
>>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>>> but it didn't seem to actually clean anything up. Is there any
>>> hope of recovery without bouncing the web application?
>>
>> The background thread is started via threadStart() which is called
>> from ContainerBase.startInternal() /
>> StandardContext.startInternal().
>>
>> So I see no hope of recovering unless the container that owns the
>> thread is stopped/started. I think that by default the thread is
>> started by Engine, so Engine needs to be bounced here.
>
> The JMX method "backgroundProcess" should call
> ManagerBase.backgroundProcess which should go ahead and expire the
> sessions. I don't see a reason why it wouldn't work. Unfortunately,
> Tomcat's Manager wasn't in debug-log mode or I would have been able to
> see what happened when I invoked backgroundProcess.
>
1. If you have Manager webapp installed, it has API to expire sessions
(on the "list" page in HTML gui)
2. I think logging level can be manipulated via JMX.
>> BTW, javadoc for ContainerBase.backgroundProcess() says "This
>> method will be invoked inside the classloading context of this
>> container." I think that is not true when it is being invoked via
>> JMX.
>
> This is ManagerBase.backgroundProcess, which does not have the same
> documentation. While ContainerBase.backgroundProcess would usually
> invoke ManagerBase.backgroundProcess, using JMX goes directly to
> ManagerBase.
OK.
> I'm still curious why invoking ManagerBase.backgroundProcess via JMX
> had no effect. I'm not sure what the "count" and
> "processExpiresFrequency" values were at the time, so my calls could
> have been rejected due to background-processing-throttling.
//ManagerBase.java
@Override
public void backgroundProcess() {
count = (count + 1) % processExpiresFrequency;
if (count == 0)
processExpires();
}
processExpiresFrequency should be readable via JMX.
Default is '6', so you have to call the method 6 times in a row.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Background thread died; no errors in log; invoking backgroundProcess
via JMX has no effect
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Konstantin,
On 6/5/14, 11:58 AM, Konstantin Kolinko wrote:
> 2014-06-05 19:19 GMT+04:00 Christopher Schultz
> <ch...@christopherschultz.net>:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>
>> All,
>>
>> One of dev instances this morning is screaming because I have
>> hundreds of active sessions. I checked, and it looks like the
>> background processor thread for the context is not running,
>> therefore no sessions are expiring.
>>
>> Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
>>
>> Here are the still-running threads:
>>
>> "catalina-exec-946" "Attach Listener" "MySQL Statement
>> Cancellation Timer" "catalina-exec-338" "catalina-exec-337"
>> "catalina-exec-5" "catalina-exec-4" "catalina-exec-3"
>> "http-nio-127.0.0.1-8217-Acceptor-0"
>> "http-nio-127.0.0.1-8217-ClientPoller-1"
>> "http-nio-127.0.0.1-8217-ClientPoller-0"
>> "ajp-bio-8215-AsyncTimeout" "ajp-bio-8215-Acceptor-0" "Memcached
>> IO over {MemcachedConnection to localhost/127.0.0.1:11211}"
>> "Abandoned connection cleanup thread" "Thread-3"
>> "NioBlockingSelector.BlockPoller-1" "GC Daemon" "Service Thread"
>> "C2 CompilerThread1" "C2 CompilerThread0" "Signal Dispatcher"
>> "Finalizer" "Reference Handler" "main" "VM Thread" "GC task
>> thread#0 (ParallelGC)" "GC task thread#1 (ParallelGC)" "VM
>> Periodic Task Thread"
>>
>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>> but it didn't seem to actually clean anything up. Is there any
>> hope of recovery without bouncing the web application?
>>
>> Also, it would be nice to get a notification if the thread is
>> dying due to some exception. I searched-though catalina.out and
>> found nothing relevant. Any suggestions?
>
> In general, dying threads are handled via
> Thread.setUncaughtExceptionHandler(...) and
> ThreadGroup.uncaughtException(...)
>
> The default behaviour in JRE it so log to System.err. So
> catalina.out is the place where I expect it to be written.
That's what I would have thought, too. I didn't see anything,
unfortunately.
> (It is not nice as it bypasses logging framework, but that is how
> it is currently).
>
> A unusual culprit is OutOfMemoryError.
I think you mean "a usual culprit" and I agree: this has happened in
the past where the background processor thread dies due to OOME.
>> I tried invoking "backgroundProcess" on my Manager bean via JMX
>> but it didn't seem to actually clean anything up. Is there any
>> hope of recovery without bouncing the web application?
>
> The background thread is started via threadStart() which is called
> from ContainerBase.startInternal() /
> StandardContext.startInternal().
>
> So I see no hope of recovering unless the container that owns the
> thread is stopped/started. I think that by default the thread is
> started by Engine, so Engine needs to be bounced here.
The JMX method "backgroundProcess" should call
ManagerBase.backgroundProcess which should go ahead and expire the
sessions. I don't see a reason why it wouldn't work. Unfortunately,
Tomcat's Manager wasn't in debug-log mode or I would have been able to
see what happened when I invoked backgroundProcess.
> BTW, javadoc for ContainerBase.backgroundProcess() says "This
> method will be invoked inside the classloading context of this
> container." I think that is not true when it is being invoked via
> JMX.
This is ManagerBase.backgroundProcess, which does not have the same
documentation. While ContainerBase.backgroundProcess would usually
invoke ManagerBase.backgroundProcess, using JMX goes directly to
ManagerBase.
I'm still curious why invoking ManagerBase.backgroundProcess via JMX
had no effect. I'm not sure what the "count" and
"processExpiresFrequency" values were at the time, so my calls could
have been rejected due to background-processing-throttling.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTkL/LAAoJEBzwKT+lPKRYSn0P/Rs3TRlD0W7YTGuYuxIbeIoc
9CODVusNETHiUU0zrY4cJwZFXhBkfJ/ev8h9+ErEIsB854KJQNJ67F1o2iNmr3Nm
qyWHldZjaYSdns3zHKhEkhyhEutIFfM8KHMDE0azc7GtKEkKNs8en8MVuBPITTWY
JYtjh8fTaxm+rfjuLp/vRRB+Oj9GYQncuLihCvWPUzWMr+sRM0/baRaWAKP3d4M6
4EC7r2Pi3HuqMbUEYt5Vh3O3x/5BOdLsU/tu616L9fGC6aiNF2w3KUIw5pECdPS7
9vIIsXhsGQZ2+Y1TpnN/lPojItTtak6hdweMNGAKberxe816YjEdTrwkeue6MiLD
J9U8ArQLI58CJCcfM2My+dyaPea53VsBKPkYbcDpO+YM1sqko9JkJTsv1fpzeKVn
NtiCQNRq7UoaIJ7u3seTXn7piJ8ahrD+FpkLXxOiLwQ7lLNKtR4b+14smwKDrI/h
gUobA92SQNgluLetnpxAvcve1sjySSVs0RN1rpw1GPTDrh/cOWIYkdiiUzZn58lf
Ro1FyotAy9JHVDCFJpcwwIzaVty/6cJpJHI+2Ukl4aMHJRjFClioDb6Gx/wBnk7I
SNiCj1R3EKxbvrTSPruHYTSdREtQHWQYpDj6odfuNtupY65K86iS4fkNTv08CYlt
y64Igz0yrJPwqCdgAPL4
=kOau
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Background thread died; no errors in log; invoking
backgroundProcess via JMX has no effect
Posted by Konstantin Kolinko <kn...@gmail.com>.
2014-06-05 19:19 GMT+04:00 Christopher Schultz <ch...@christopherschultz.net>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> All,
>
> One of dev instances this morning is screaming because I have hundreds
> of active sessions. I checked, and it looks like the background
> processor thread for the context is not running, therefore no sessions
> are expiring.
>
> Tomcat 7.0.52, Debian Linux x64, Oracle Java 1.7.0_55
>
> Here are the still-running threads:
>
> "catalina-exec-946"
> "Attach Listener"
> "MySQL Statement Cancellation Timer"
> "catalina-exec-338"
> "catalina-exec-337"
> "catalina-exec-5"
> "catalina-exec-4"
> "catalina-exec-3"
> "http-nio-127.0.0.1-8217-Acceptor-0"
> "http-nio-127.0.0.1-8217-ClientPoller-1"
> "http-nio-127.0.0.1-8217-ClientPoller-0"
> "ajp-bio-8215-AsyncTimeout"
> "ajp-bio-8215-Acceptor-0"
> "Memcached IO over {MemcachedConnection to localhost/127.0.0.1:11211}"
> "Abandoned connection cleanup thread"
> "Thread-3"
> "NioBlockingSelector.BlockPoller-1"
> "GC Daemon"
> "Service Thread"
> "C2 CompilerThread1"
> "C2 CompilerThread0"
> "Signal Dispatcher"
> "Finalizer"
> "Reference Handler"
> "main"
> "VM Thread"
> "GC task thread#0 (ParallelGC)"
> "GC task thread#1 (ParallelGC)"
> "VM Periodic Task Thread"
>
> I tried invoking "backgroundProcess" on my Manager bean via JMX but it
> didn't seem to actually clean anything up. Is there any hope of
> recovery without bouncing the web application?
>
> Also, it would be nice to get a notification if the thread is dying
> due to some exception. I searched-though catalina.out and found
> nothing relevant. Any suggestions?
In general, dying threads are handled via
Thread.setUncaughtExceptionHandler(...)
and
ThreadGroup.uncaughtException(...)
The default behaviour in JRE it so log to System.err. So catalina.out
is the place where I expect it to be written.
(It is not nice as it bypasses logging framework, but that is how it
is currently).
A unusual culprit is OutOfMemoryError.
> I tried invoking "backgroundProcess" on my Manager bean via JMX but it
> didn't seem to actually clean anything up. Is there any hope of
> recovery without bouncing the web application?
The background thread is started via threadStart() which is called
from ContainerBase.startInternal() / StandardContext.startInternal().
So I see no hope of recovering unless the container that owns the
thread is stopped/started. I think that by default the thread is
started by Engine, so Engine needs to be bounced here.
BTW, javadoc for ContainerBase.backgroundProcess() says "This method
will be invoked inside the classloading context of this container." I
think that is not true when it is being invoked via JMX.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org