You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mike Noordermeer <mi...@normi.net> on 2016/08/10 10:15:30 UTC
Tomcat 8.5 NIO w/ SSL Windows CPU issues
Hi,
After an upgrade to Tomcat 8.5, we are experiencing an issue where
Tomcat starts generating a high CPU load (100%), probably after an
HTTP network scan. The bug seems to be related to Windows, NIO and
possibly SSL. I have a Yourkit dump and several thread dumps that show
the issue, and was wondering if anyone is interested in this, and if
we can gather any extra information to help debug this issue.
Setup: Windows 2k8r2, Tomcat 8.5.4, Java 8u102, NIO HTTP and NIO JSSE
HTTPS connector.
Out of nothing, Tomcat starts using 100% CPU. I made some thread
dumps, available here:
https://gist.github.com/MikeN123/f4a85f09231cfda7a9e632b64f27dcdc
https://gist.github.com/MikeN123/7dfe17ae95b8d516d86e0d7126cbaa02
https://gist.github.com/MikeN123/750da8580e04e0498f70b81dbd1a5c52
https://gist.github.com/MikeN123/2e83307b7c1216339d4fa73b30c02f1a
https://gist.github.com/MikeN123/8850ef2a60d39a4dc140b2d8fef18c3f
I also have some Yourkit stats available, but as these may contain
confidential information, I won't share them in public. Basically,
what we see is that the thread https-jsse-nio-443-ClientPoller-0 is
continuously runnable and using CPU on
sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(), and various other
https-jsse-nio-443-exec threads are waiting (parked) or running. These
threads together take up all the CPU. A Yourkit thread view showing
the issue starting around 11:02: https://dl.eveoh.nl/yc_fal.png
We _suspect_ the issue is triggered by an HTTP scan, which generates
the following requests in the access log, but we are still trying to
confirm this: https://gist.github.com/MikeN123/581d1f17aae100f06b8c65b86870a64a
Also, we are trying to confirm whether or not NIO2 shows the same behaviour.
The behaviour seems to be the same as in this tomcat-users thread:
https://mail-archives.apache.org/mod_mbox/tomcat-users/201604.mbox/%3CCAE-ydNF84pnoX2tP8BJ4vQisabgycP0y2vpnmjNhddz9+BKp=w@mail.gmail.com%3E
A similar issue is mentioned for some other products, but I'm not sure
if there's a relation:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=357240
https://developer.jboss.org/thread/240618?start=0&tstart=0
https://github.com/netty/netty/issues/3857
Our next steps are:
- Switching the production site to NIO2, to see if that fixes the issue
- Checking if we can reproduce the issue by triggering the HTTP
vulnerability scan manually
Any ideas or requests for more information are more than welcome.
Regards,
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 8.5 NIO w/ SSL Windows CPU issues
Posted by Mike Noordermeer <mi...@normi.net>.
Hi,
> Are you fronting with a web server/reverse proxy? Those "-" requests
> looks suspiciously like the kinds of requests that Apache httpd makes
> to itself to verify that worker threads are still available for
> certain things.
No, there is no web server or proxy in front of Tomcat.
> I'm curious... why are the requests coming from "10.xxx"... isn't that
> within your own network? Shouldn't you KNOW what that stuff is?
It's somewhere in the customer's network. Not entirely sure what it
is, the sysadmin of that box will be back tomorrow, it is probably
some kind of security scan. When he's back we will be able to check if
the problem is indeed caused by the scanner. It's only a couple of
requests, so it shouldn't trigger a high CPU use (would be a DoS issue
in Tomcat/Java), but if it's reproducible it will at least allow us to
better pinpoint the problem. The problem hasn't resurfaced after a
switch to NIO2, so it seems NIO specific.
Kind regards,
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 8.5 NIO w/ SSL Windows CPU issues
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
On 10.08.2016 17:46, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Mike,
>
> On 8/10/16 6:15 AM, Mike Noordermeer wrote:
>> Hi,
>>
>> After an upgrade to Tomcat 8.5, we are experiencing an issue where
>> Tomcat starts generating a high CPU load (100%), probably after an
>> HTTP network scan. The bug seems to be related to Windows, NIO and
>> possibly SSL. I have a Yourkit dump and several thread dumps that
>> show the issue, and was wondering if anyone is interested in this,
>> and if we can gather any extra information to help debug this
>> issue.
>>
>> Setup: Windows 2k8r2, Tomcat 8.5.4, Java 8u102, NIO HTTP and NIO
>> JSSE HTTPS connector.
>>
>> Out of nothing, Tomcat starts using 100% CPU. I made some thread
>> dumps, available here:
>>
>> https://gist.github.com/MikeN123/f4a85f09231cfda7a9e632b64f27dcdc
>> https://gist.github.com/MikeN123/7dfe17ae95b8d516d86e0d7126cbaa02
>> https://gist.github.com/MikeN123/750da8580e04e0498f70b81dbd1a5c52
>> https://gist.github.com/MikeN123/2e83307b7c1216339d4fa73b30c02f1a
>> https://gist.github.com/MikeN123/8850ef2a60d39a4dc140b2d8fef18c3f
>>
>> I also have some Yourkit stats available, but as these may contain
>> confidential information, I won't share them in public. Basically,
>> what we see is that the thread https-jsse-nio-443-ClientPoller-0
>> is continuously runnable and using CPU on
>> sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(), and various
>> other https-jsse-nio-443-exec threads are waiting (parked) or
>> running. These threads together take up all the CPU. A Yourkit
>> thread view showing the issue starting around 11:02:
>> https://dl.eveoh.nl/yc_fal.png
>>
>> We _suspect_ the issue is triggered by an HTTP scan, which
>> generates the following requests in the access log, but we are
>> still trying to confirm this:
>> https://gist.github.com/MikeN123/581d1f17aae100f06b8c65b86870a64a
>>
>> Also, we are trying to confirm whether or not NIO2 shows the same
>> behaviour.
>>
>> The behaviour seems to be the same as in this tomcat-users thread:
>> https://mail-archives.apache.org/mod_mbox/tomcat-users/201604.mbox/%3C
> CAE-ydNF84pnoX2tP8BJ4vQisabgycP0y2vpnmjNhddz9+BKp=w@mail.gmail.com%3E
>>
>> A similar issue is mentioned for some other products, but I'm not
>> sure if there's a relation:
>>
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=357240
>> https://developer.jboss.org/thread/240618?start=0&tstart=0
>> https://github.com/netty/netty/issues/3857
>>
>> Our next steps are:
>>
>> - Switching the production site to NIO2, to see if that fixes the
>> issue - Checking if we can reproduce the issue by triggering the
>> HTTP vulnerability scan manually
>>
>> Any ideas or requests for more information are more than welcome.
>
> Are you fronting with a web server/reverse proxy? Those "-" requests
> looks suspiciously like the kinds of requests that Apache httpd makes
> to itself to verify that worker threads are still available for
> certain things.
>
> Maybe that's a way that HTTP scanners are trying to avoid detection:
> by looking like "normal" stuff in the logs.
>
> I'm curious... why are the requests coming from "10.xxx"... isn't that
> within your own network? Shouldn't you KNOW what that stuff is?
In-house webserver monitoring software ? (Nagios e.g.)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 8.5 NIO w/ SSL Windows CPU issues
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mike,
On 8/10/16 6:15 AM, Mike Noordermeer wrote:
> Hi,
>
> After an upgrade to Tomcat 8.5, we are experiencing an issue where
> Tomcat starts generating a high CPU load (100%), probably after an
> HTTP network scan. The bug seems to be related to Windows, NIO and
> possibly SSL. I have a Yourkit dump and several thread dumps that
> show the issue, and was wondering if anyone is interested in this,
> and if we can gather any extra information to help debug this
> issue.
>
> Setup: Windows 2k8r2, Tomcat 8.5.4, Java 8u102, NIO HTTP and NIO
> JSSE HTTPS connector.
>
> Out of nothing, Tomcat starts using 100% CPU. I made some thread
> dumps, available here:
>
> https://gist.github.com/MikeN123/f4a85f09231cfda7a9e632b64f27dcdc
> https://gist.github.com/MikeN123/7dfe17ae95b8d516d86e0d7126cbaa02
> https://gist.github.com/MikeN123/750da8580e04e0498f70b81dbd1a5c52
> https://gist.github.com/MikeN123/2e83307b7c1216339d4fa73b30c02f1a
> https://gist.github.com/MikeN123/8850ef2a60d39a4dc140b2d8fef18c3f
>
> I also have some Yourkit stats available, but as these may contain
> confidential information, I won't share them in public. Basically,
> what we see is that the thread https-jsse-nio-443-ClientPoller-0
> is continuously runnable and using CPU on
> sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(), and various
> other https-jsse-nio-443-exec threads are waiting (parked) or
> running. These threads together take up all the CPU. A Yourkit
> thread view showing the issue starting around 11:02:
> https://dl.eveoh.nl/yc_fal.png
>
> We _suspect_ the issue is triggered by an HTTP scan, which
> generates the following requests in the access log, but we are
> still trying to confirm this:
> https://gist.github.com/MikeN123/581d1f17aae100f06b8c65b86870a64a
>
> Also, we are trying to confirm whether or not NIO2 shows the same
> behaviour.
>
> The behaviour seems to be the same as in this tomcat-users thread:
> https://mail-archives.apache.org/mod_mbox/tomcat-users/201604.mbox/%3C
CAE-ydNF84pnoX2tP8BJ4vQisabgycP0y2vpnmjNhddz9+BKp=w@mail.gmail.com%3E
>
> A similar issue is mentioned for some other products, but I'm not
> sure if there's a relation:
>
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=357240
> https://developer.jboss.org/thread/240618?start=0&tstart=0
> https://github.com/netty/netty/issues/3857
>
> Our next steps are:
>
> - Switching the production site to NIO2, to see if that fixes the
> issue - Checking if we can reproduce the issue by triggering the
> HTTP vulnerability scan manually
>
> Any ideas or requests for more information are more than welcome.
Are you fronting with a web server/reverse proxy? Those "-" requests
looks suspiciously like the kinds of requests that Apache httpd makes
to itself to verify that worker threads are still available for
certain things.
Maybe that's a way that HTTP scanners are trying to avoid detection:
by looking like "normal" stuff in the logs.
I'm curious... why are the requests coming from "10.xxx"... isn't that
within your own network? Shouldn't you KNOW what that stuff is?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJXq0xpAAoJEBzwKT+lPKRYu/sP/RdIRuBnSq36wHr8eYj6IAc9
xJmlcjdfBB4CL7DYwDvH+fddDb/W8k48ksQIk/nhvrqf0mP7V7g21L9pS3Cl/ki4
i/pKR6WhxPEkcKGZh7uxlD6bPUpJCVcvDX0W8jfl+yXXX5eJKVWwrfJHxZ7i4hEL
c8Vz/JUyXfAATWbfkiBGRhrcSqA3QG9WD1D6XZxAl6saJgqUD2tn7CbXJ+XG6GZB
563qD/fBg20FG5xq5atXhnjAl4icmhIEcaszjsgNb7cwzwPKV2PBcgF+V6ObGvHA
+5v0by0jGpaX6I5wozjVMIOpjdcQLFApimdDS4yBaqP400jKrFsbwVVjYn1E08oK
n9lxW2yfzC1l9AmbnHa2OcpRSyFVV9XyZdgkElVETz8XQbJu+E18gK9eTKufweYA
PDUVmNwXg/WaCNpZtuG8kvgsP93v4mfTzsru9I5ktMSdSti5cJPtrkHGnZicBZNB
5oWQzgtc38exUyJg9WKalAZMNiAR1WlH4mxCCDmS8TAUd0JzKexRRsQYRDKYQCJY
SU3iW5zvj5n1dcloMdyx1BbP6pGIK/FvP9Jxm4oKhDfN4WnrFHa/eYWam1B7lpO/
sgu7KJe815zgGOZS8H5cstegY2SAsi48zG1JLQut+WZE/2fBkozZVqdj/D/ybd4n
WPi5Lf4v5RcbL5w1f6OB
=19NA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org