You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mark Thomas <ma...@apache.org> on 2017/03/23 08:43:31 UTC
Re: Tomcat Not Responding
On 23/03/17 05:07, Igal @ Lucee.org wrote:
> Hi Mark,
>
> On 3/22/2017 1:00 PM, Mark Thomas wrote:
>> On 22/03/17 16:20, Igal @ Lucee.org wrote:
>>> Looks like all the threads are waiting on the same lock?
>> Agreed. But I don't see anything holding that lock. That looks like a
>> JVM bug to me.
>
> Thank you for replying. I managed to "do" it again. This time on
> Tomcat 8.5.11 so I modified the subject line slightly. But this time I
> took both a thread dump (attached) and a heap dump.
>
> I should mention that my code utilizes JSR-356 WebSockets, I'm not sure
> if that's related, but it seems to be a factor in reproducing the issue.
>
> I will try to create a reduced test case.
>
> I am hoping that the screenshots below, showing references to and from
> the lock object at 0x00000006c0dbc1b8, can give you some more
> information that may help to diagnose this issue, or at least give some
> ideas about what to look for next:
I took another look and those threads are in the normal state you'd
expect for threads in the pool waiting for work.
If Tomcat is unresponsive, that would suggest a problem with either the
Acceptor or the Poller but both of those threads look normal to.
You are going to need to do some more debugging.
Ideally, you need to configure your favourite IDE to do remote debugging
and then once the problem occurs, step through the code as a single
connection is handled and see where things go wrong.
Based on what I see in the thread dumps (all normal) I'd also check that
your test is doing what it is meant to.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Not Responding (Resolved)
Posted by "Igal @ Lucee.org" <ig...@lucee.org>.
The only thing I saw in stderr were NPE stack traces from the servlet
I'm running which did not seem to include any Tomcat code.
Keep in mind that I'm running Tomcat 8.5.12 so a more current source
repo is available at
https://github.com/apache/tomcat85/
With the file that you cited at
https://github.com/apache/tomcat85/blob/trunk/java/org/apache/catalina/core/StandardServer.java
Igal Sapir
Lucee Core Developer
Lucee.org <http://lucee.org/>
On 3/28/2017 7:14 AM, Suvendu Sekhar Mondal wrote:
> Well, I took thread dumps on couple of Tomcat JVMs(which were not
> busy...JVMs on my system) and for all of them it showed thread opened
> socket and accepting connection(StandardServer.java:446):
>
> at java.net.DualStackPlainSocketImpl.accept0(Native Method)
> at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
> at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
> at java.net.PlainSocketImpl.accept(Unknown Source)
> - locked <0x00000000fbf67198> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(Unknown Source)
> at java.net.ServerSocket.accept(Unknown Source)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
>
> But I don't see that for problematic JVM. It was stuck before
> that(StandardServer.java:427) step.
>
> at java.lang.Thread.sleep(Native Method)
> at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
>
> That's why I suspect JVM was not properly up and running. In fact, I
> never saw a main thread in TIMED_WAITING waiting state. Again, I might
> be wrong. :)
>
> Thanks!
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: Tomcat Not Responding (Resolved)
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Well, I took thread dumps on couple of Tomcat JVMs(which were not
busy...JVMs on my system) and for all of them it showed thread opened
socket and accepting connection(StandardServer.java:446):
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0x00000000fbf67198> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
But I don't see that for problematic JVM. It was stuck before
that(StandardServer.java:427) step.
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
That's why I suspect JVM was not properly up and running. In fact, I
never saw a main thread in TIMED_WAITING waiting state. Again, I might
be wrong. :)
Thanks!
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Not Responding (Resolved)
Posted by Mark Thomas <ma...@apache.org>.
On 28/03/17 13:50, Suvendu Sekhar Mondal wrote:
> Sorry for late "chime in".
>
> It seems that your Tomcat JVM was not up properly. From both thread
> dumps showing main thread in TIMED_WAITING waiting state not in
> runnable state :
Your analysis is incorrect. That is normal and expected for the main
thread (it is waiting for a shutdown signal).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Not Responding (Resolved)
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Sorry for late "chime in".
It seems that your Tomcat JVM was not up properly. From both thread
dumps showing main thread in TIMED_WAITING waiting state not in
runnable state :
"main" #1 prio=5 os_prio=0 tid=0x0000000001198000 nid=0x2340 waiting
on condition [0x000000000132e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:427)
at org.apache.catalina.startup.Catalina.await(Catalina.java:743)
at org.apache.catalina.startup.Catalina.start(Catalina.java:689)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:355)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:495)
Locked ownable synchronizers:
- None
I went by the source code(which is little bit outdated):
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.tomcat/tomcat-catalina/8.0.24/org/apache/catalina/core/StandardServer.java/
Did you check stderr/stdout log? It should log some error.
Here is snippet of main thread from running Tomcat JVM:
"main" #1 prio=5 os_prio=0 tid=0x0000000001084000 nid=0x2008 runnable
[0x000000000107e000]
java.lang.Thread.State: RUNNABLE
at java.net.DualStackPlainSocketImpl.accept0(Native Method)
at java.net.DualStackPlainSocketImpl.socketAccept(Unknown Source)
at java.net.AbstractPlainSocketImpl.accept(Unknown Source)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked <0x00000000fbf67198> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:446)
at org.apache.catalina.startup.Catalina.await(Catalina.java:713)
at org.apache.catalina.startup.Catalina.start(Catalina.java:659)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:351)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:485)
Locked ownable synchronizers:
- None
Thanks!
Suvendu
On Sat, Mar 25, 2017 at 2:51 AM, Igal @ Lucee.org <ig...@lucee.org> wrote:
> Chris,
>
> On 3/24/2017 2:13 PM, Christopher Schultz wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Igal,
>>
>> On 3/24/17 1:22 PM, Igal @ Lucee.org wrote:
>>>
>>> I've traced the issue to an NPE thrown from my servlet. I patched
>>> it (https://github.com/lucee/Lucee/commit/0f30a7ef) and now it
>>> works fine.
>>
>> That's good to know, but an NPE thrown from an application shouldn't
>> be able to lock-up Tomcat. Can you reproduce this using an SSCCE[1]?
>
>
> I agree. That's the reason I created a simple html file and tested it, as
> those do not go through the Lucee servlet.
>
> I will try to produce a reduced test case, but not sure how simple that
> would be.
>
>
> Igal
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Not Responding (Resolved)
Posted by "Igal @ Lucee.org" <ig...@lucee.org>.
Chris,
On 3/24/2017 2:13 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Igal,
>
> On 3/24/17 1:22 PM, Igal @ Lucee.org wrote:
>> I've traced the issue to an NPE thrown from my servlet. I patched
>> it (https://github.com/lucee/Lucee/commit/0f30a7ef) and now it
>> works fine.
> That's good to know, but an NPE thrown from an application shouldn't
> be able to lock-up Tomcat. Can you reproduce this using an SSCCE[1]?
I agree. That's the reason I created a simple html file and tested it,
as those do not go through the Lucee servlet.
I will try to produce a reduced test case, but not sure how simple that
would be.
Igal
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Not Responding (Resolved)
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Igal,
On 3/24/17 1:22 PM, Igal @ Lucee.org wrote:
> On 3/23/2017 1:43 AM, Mark Thomas wrote:
>> If Tomcat is unresponsive, that would suggest a problem with
>> either the Acceptor or the Poller but both of those threads look
>> normal to.
>>
>> You are going to need to do some more debugging.
>>
>> Ideally, you need to configure your favourite IDE to do remote
>> debugging and then once the problem occurs, step through the code
>> as a single connection is handled and see where things go wrong.
>
> I've traced the issue to an NPE thrown from my servlet. I patched
> it (https://github.com/lucee/Lucee/commit/0f30a7ef) and now it
> works fine.
That's good to know, but an NPE thrown from an application shouldn't
be able to lock-up Tomcat. Can you reproduce this using an SSCCE[1]?
- -chris
[1] http://sscce.org/
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJY1YvyAAoJEBzwKT+lPKRYBrUP+wQKAw4Z/Pj1m77BHXikXBwG
zHqt0dHai2IXOMGI3uDcyugsO9f/+4LqhZvMoD7M4gsO9AgjO9lyhWI/CLzLj75Q
2UK68MmeTKyzRPTyEgmFzfwu3YlH8ymRQ84Qg9hS+ICyAmtCQNGvDQvzqnminKSG
oNgt+cCmMwPZxLpNSZF4iuFiIvu++CwuynMbn5x3mscfc3ZcdjrHfAVWJwgGDStG
XxPE/NpBIwEOZ5SpWCu0rpieLPuuDa/g2Orp4wlSn/It4kNTtr3LgznyVUBMHodx
R1CRIueV08eW0TVhqmN+7uwzRP8es1o2NuQX0sz6123DRBOV4abZCpzJjG+ul4jf
qFW+uZw8yD9rbAS8ttZlgO1SasjV9BLsQ+YY4IejVubfZbS+CR08VNO3fPEvSVm6
huw1W1EsjPDeBWa8khvRqv/aFEHkwt0/bC52fdk8DwUI7hqgr7xaMER+nwW7LqcS
6AbjrQhl1wfklBH8zOwLZUXc3PRA/+qCq4W+Q8mzleQ/uW2yC151zyAz9VZYAakh
gQ7mh+VJsSCLvxVUvZFOSBxk+06HVh1dSOgHoIUat4OqYz7pDA7OEidgpfLYK6/z
hps8alYBJnDLcORG8/OtApvrPKz6Ysskuw2nY5tOdClH8a/rvhS3mA64e9zRsZ9+
UU9JHytsNlKFO0a8W1dY
=0ChH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Not Responding (Resolved)
Posted by "Igal @ Lucee.org" <ig...@lucee.org>.
On 3/23/2017 1:43 AM, Mark Thomas wrote:
> If Tomcat is unresponsive, that would suggest a problem with either the
> Acceptor or the Poller but both of those threads look normal to.
>
> You are going to need to do some more debugging.
>
> Ideally, you need to configure your favourite IDE to do remote debugging
> and then once the problem occurs, step through the code as a single
> connection is handled and see where things go wrong.
I've traced the issue to an NPE thrown from my servlet. I patched it
(https://github.com/lucee/Lucee/commit/0f30a7ef) and now it works fine.
Thank you,
Igal Sapir
Lucee Core Developer
Lucee.org <http://lucee.org/>