You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2017/01/12 12:17:17 UTC
[Bug 60578] New: Server CPU maxed out (100% per core) randomly after
a few hours
https://bz.apache.org/bugzilla/show_bug.cgi?id=60578
Bug ID: 60578
Summary: Server CPU maxed out (100% per core) randomly after a
few hours
Product: Tomcat 8
Version: 8.0.14
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Connectors
Assignee: dev@tomcat.apache.org
Reporter: hauser@acm.org
Target Milestone: ----
On Debian stable since a few days the tomcat all of a sudden goes to next to
100% CPU.
Not much traffic is seen on the firewall .
Is this related to Bug 57544 ?
For the amount of cpu used, tomcat is astonishingly responsive.
Some have speculated that this is a subtle kind of DOS attack and with a
kill -QUIT
I get dozens of the two threads I don't before the cpu went high:
"http-nio-171.24.2.105-8443-exec-14" #13579 daemon prio=5 os_prio=0
tid=0x00007fa68c040000 nid=0x2352 runnable [0x00007fa614461000]
java.lang.Thread.State: RUNNABLE
at
org.apache.coyote.http11.AbstractInputBuffer.nextRequest(AbstractInputBuffer.java:244)
at
org.apache.coyote.http11.AbstractNioInputBuffer.nextRequest(AbstractNioInputBuffer.java:151)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1152)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:658)
at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:222)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1566)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1523)
- locked <0x0000000095c30eb8> (a
org.apache.tomcat.util.net.SecureNioChannel)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
"http-nio-171.24.2.105-ClientPoller-1" #112 daemon prio=5 os_prio=0
tid=0x00007fa6b4e84000 nid=0x45db runnable [0x00007fa61aff8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000866957c0> (a sun.nio.ch.Util$3)
- locked <0x00000000866957b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000086695688> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1050)
at java.lang.Thread.run(Thread.java:745)
The same version of tomcat was in use without the phenomenon for weeks
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 60578] Server CPU maxed out (100% per core) randomly after a
few hours
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60578
--- Comment #3 from Ralf Hauser <ha...@acm.org> ---
thanks, the backport 8.5.9 appears to solve the problem (albeit not that long
observation period).
One side-effect was that the Bug 60126 came.
<<The code of method _jspService(HttpServletRequest, HttpServletResponse) is
exceeding the 65535 bytes limit>>
Adding
<init-param>
<param-name>mappedfile</param-name>
<param-value>false</param-value>
</init-param>
to [Tomcat_Home]/conf/web.xml as per
https://www.assetbank.co.uk/support/documentation/knowledge-base/byte-limit-exceeded-error/
solved that for me, but this might probably not work for everyone with the same
problem
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 60578] Server CPU maxed out (100% per core) randomly after a
few hours
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60578
--- Comment #4 from Mark Thomas <ma...@apache.org> ---
Note the root cause of this in Debian, Ubuntu etc. was back-porting the
security fix for CVE-2016-6816 without back-porting the 57544 fix. This made it
trivial to trigger the loop described in bug 57544.
Without the back-port of the CVE-2016-6816 the loop described in bug 57544 was
significantly harder to trigger. The root cause of 57544 has not been
identified. It may have been user triggered but it may also have been triggered
by an application bug.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 60578] Server CPU maxed out (100% per core) randomly after a
few hours
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60578
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |DUPLICATE
--- Comment #1 from Mark Thomas <ma...@apache.org> ---
As far as I can tell from
http://metadata.ftp-master.debian.org/changelogs/main/t/tomcat8/tomcat8_8.0.14-1+deb8u6_changelog
the fix for BZ57544 was not back-ported.
You'll need to contact the Debian maintainers for more info.
*** This bug has been marked as a duplicate of bug 57544 ***
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 60578] Server CPU maxed out (100% per core) randomly after a
few hours
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60578
--- Comment #2 from Emmanuel Bourg <eb...@apache.org> ---
I'll take care of backporting the fix to Debian. In the meantime, you can
install the latest version of the tomcat8 package from the jessie-backports
repository. You'll get the latest 8.5.9 version with the fix included.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 60578] Server CPU maxed out (100% per core) randomly after a
few hours
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60578
1102804635@qq.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Target Milestone|---- |---
Product|Tomcat 8 |Tomcat 7
Version|8.0.14 |7.0.93
Component|Connectors |Connectors
OS|Linux |Windows Server 2008
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 60578] Server CPU maxed out (100% per core) randomly after a
few hours
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60578
--- Comment #5 from Jerome Terry <je...@gmail.com> ---
I have experienced what appears to be the same issue on Ubuntu 14.04 with
Tomcat 7.0.52. Here's a link to tweets containing the diagnostics I performed.
https://twitter.com/jeromeleoterry/status/831865811962908672
In my use case, a Nessus scan on ports 8080 and 8009 was triggering the CPU to
get maxed out. I was able to reproduce this issue in a QA environment with no
load applied to the tomcat, then triggered an Nessus scan. Nessus scan with
only the HTTPS connector enabled didn't trigger the CPU staking at 100%.
I ran strace and the bulk of the time was being spent in futex. I also ran
Linux perf, and AbstractHttp11Processor.process was consuming 49.91% of CPU,
while AbstractInputBuffer.nextRequest was consuming 50.06% of the CPU.
In Catalina.out, I saw the error messages "Invalid message received with
signature" and "Error parsing HTTP request header".
This is a nasty one. A security scan on port 8080 or 8009 can trigger all cores
to max out, which is a simple way of doing a denial of service attack.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org