You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by dharmon <do...@gmail.com> on 2018/04/06 21:10:21 UTC
High CPU usage with Utf8Decoder in 8.5.23
environment:
tomcat 8.5.23
CentOS 6.9
jdk8u162 b12
I recently upgraded from tomcat 7.0.61/jdk8u92 to tomcat 8.5.23/jdk8u162.
Our application war file on tomcat7 has cpu >75% idle per vmstat several
hours after tomcat start. Same war file on same server with tomcat8.5.23 has
cpu 0% idle per vmstat several hours after restart. Same server with similar
request load. Server has 4 CPUs. War file processes https REST web service
requests.
I used jvisualvm to perform CPU profiling and the method with most self time
is org.apache.tomcat.util.buf.Utf8Decoder.decodeHasArray. jstack shows there
are 12 https-jsse-nio-8443-exec-### threads always in runnable state[1].
They have been in this state for several hours. It appears as if it's stuck
in infinite loop. Using the thread's nid and lsof, I believe the tcp/https
connection associated with these threads are no longer
connected/established.
Any known issues that could be causing this problem? I also tried using the
NIO2 connector with same issue.
I'm on a closed network, thus unable to show jvisualvm screen shots or
verbose stack traces.
[1]
java.lang.Thread.State: RUNNABLE
at
org.apache.tomcat.util.buf.Utf8Decoder.decodeHasArray(Utf8Decoder.java:159)
at org.apache.tomcat.util.buf.Utf8Decoder.decodeLoop(Utf8Decoder.java:78)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:579)
at org.apache.tomcat.util.buf.B2CConverter.convert(B2CConverter.java:273)
at
org.apache.catalina.connector.InputBuffer.realReadChars(InputBuffer.java:426)
at
org.apache.catalina.connector.InputBuffer.checkCharBufferEof(InputBuffer.java:652)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:468)
at org.apache.catalina.connector.CoyoteReader.read(CoyoteReader.java:101)
<groovy classes>
<my application code with is attempting to read http request>
--
Sent from: http://tomcat.10.x6.nabble.com/Tomcat-User-f1968778.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU usage with Utf8Decoder in 8.5.23
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 4/9/18 11:11 AM, Mark Thomas wrote:
> On 09/04/18 14:20, Christopher Schultz wrote:
>> Mark,
>>
>> On 4/8/18 6:39 PM, Mark Thomas wrote:
>>> On 08/04/2018 21:29, Christopher Schultz wrote:
>>>
>>> <snip/>
>>>
>>>> Does Tomcat do its own UTF-8 decoding because the JVM doesn't have a
>>>> facility to convert from ByteBuffer to CharBuffer? That seems like
>>>> something the JVM really should be providing...
>>>
>>> No. It does it because the JRE UTF-8 decoder is buggy. Some bugs were
>>> fixed in Java 8 and the rest in Java 9 so we need this decoder until
>>> Java 9 is the minimum.
>>
>> Gotcha. Are there any known remaining bugs in the Java 9 implementation?
>
> Not that I am aware of.
>
>> If not, should we go ahead and use the JVM-provided UTF-8 decoder when
>> we detect a suitable version of Java? Or is it simply not worth it?
>
> We could. From memory there were some performance issues as well but
> that would need to be retested to see if they still apply.
>
>> Sadly, the OpenJDK license is GPL, so we can't simply use their code in
>> Tomcat. :(
>
> But the Harmony implementation is ALv2 so we just used that (and fixed
> the additional bugs). :)
Right, but Harmony is dead, Jim. Er, Mark.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU usage with Utf8Decoder in 8.5.23
Posted by Mark Thomas <ma...@apache.org>.
On 09/04/18 14:20, Christopher Schultz wrote:
> Mark,
>
> On 4/8/18 6:39 PM, Mark Thomas wrote:
>> On 08/04/2018 21:29, Christopher Schultz wrote:
>>
>> <snip/>
>>
>>> Does Tomcat do its own UTF-8 decoding because the JVM doesn't have a
>>> facility to convert from ByteBuffer to CharBuffer? That seems like
>>> something the JVM really should be providing...
>>
>> No. It does it because the JRE UTF-8 decoder is buggy. Some bugs were
>> fixed in Java 8 and the rest in Java 9 so we need this decoder until
>> Java 9 is the minimum.
>
> Gotcha. Are there any known remaining bugs in the Java 9 implementation?
Not that I am aware of.
> If not, should we go ahead and use the JVM-provided UTF-8 decoder when
> we detect a suitable version of Java? Or is it simply not worth it?
We could. From memory there were some performance issues as well but
that would need to be retested to see if they still apply.
> Sadly, the OpenJDK license is GPL, so we can't simply use their code in
> Tomcat. :(
But the Harmony implementation is ALv2 so we just used that (and fixed
the additional bugs). :)
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU usage with Utf8Decoder in 8.5.23
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 4/8/18 6:39 PM, Mark Thomas wrote:
> On 08/04/2018 21:29, Christopher Schultz wrote:
>
> <snip/>
>
>> Does Tomcat do its own UTF-8 decoding because the JVM doesn't have a
>> facility to convert from ByteBuffer to CharBuffer? That seems like
>> something the JVM really should be providing...
>
> No. It does it because the JRE UTF-8 decoder is buggy. Some bugs were
> fixed in Java 8 and the rest in Java 9 so we need this decoder until
> Java 9 is the minimum.
Gotcha. Are there any known remaining bugs in the Java 9 implementation?
If not, should we go ahead and use the JVM-provided UTF-8 decoder when
we detect a suitable version of Java? Or is it simply not worth it?
Sadly, the OpenJDK license is GPL, so we can't simply use their code in
Tomcat. :(
> The issue is that incorrect decoding can lead to 'unexpected' behaviour
> when parsing URLs (read some form of security vulnerability).
Ack.
Thanks,
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU usage with Utf8Decoder in 8.5.23
Posted by Mark Thomas <ma...@apache.org>.
On 08/04/2018 21:29, Christopher Schultz wrote:
<snip/>
> Does Tomcat do its own UTF-8 decoding because the JVM doesn't have a
> facility to convert from ByteBuffer to CharBuffer? That seems like
> something the JVM really should be providing...
No. It does it because the JRE UTF-8 decoder is buggy. Some bugs were
fixed in Java 8 and the rest in Java 9 so we need this decoder until
Java 9 is the minimum.
The issue is that incorrect decoding can lead to 'unexpected' behaviour
when parsing URLs (read some form of security vulnerability).
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU usage with Utf8Decoder in 8.5.23
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 4/7/18 6:57 AM, Mark Thomas wrote:
> On 06/04/18 22:10, dharmon wrote:
>> environment:
>> tomcat 8.5.23
>> CentOS 6.9
>> jdk8u162 b12
>>
>> I recently upgraded from tomcat 7.0.61/jdk8u92 to tomcat 8.5.23/jdk8u162.
>> Our application war file on tomcat7 has cpu >75% idle per vmstat several
>> hours after tomcat start. Same war file on same server with tomcat8.5.23 has
>> cpu 0% idle per vmstat several hours after restart. Same server with similar
>> request load. Server has 4 CPUs. War file processes https REST web service
>> requests.
>>
>> I used jvisualvm to perform CPU profiling and the method with most self time
>> is org.apache.tomcat.util.buf.Utf8Decoder.decodeHasArray. jstack shows there
>> are 12 https-jsse-nio-8443-exec-### threads always in runnable state[1].
>> They have been in this state for several hours. It appears as if it's stuck
>> in infinite loop. Using the thread's nid and lsof, I believe the tcp/https
>> connection associated with these threads are no longer
>> connected/established.
>>
>> Any known issues that could be causing this problem? I also tried using the
>> NIO2 connector with same issue.
>
> None I am aware of. I wouldn't expect the connector to make any difference.
>
>> I'm on a closed network, thus unable to show jvisualvm screen shots or
>> verbose stack traces.
> I've taken a good look at the relevant code and nothing immediately
> jumps out as a potential cause of this.
>
> It sounds as if this is fairly easy to reproduce so I would suggest
> configuring your Tomcat instance to support remote debugging and when it
> happens again step through the relevant code to see why /exactly where
> it is looping. The aim is to provide us with enough information to
> ideally reproduce it but - as a minimum - provide an explanation that
> can be used to support a change to fix it.
+1
This has "edge case" written all over it. I suspect once we capture a
stream/string being read/converted, we will find that something like
Java's double-precision bug[1] where the decoder oscillates between two
states.
Does Tomcat do its own UTF-8 decoding because the JVM doesn't have a
facility to convert from ByteBuffer to CharBuffer? That seems like
something the JVM really should be providing...
Most of the code in that method is fairly old. Has the Java API caught
up with us, yet?
-chris
[1]
http://www.exploringbinary.com/java-hangs-when-converting-2-2250738585072012e-308/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU usage with Utf8Decoder in 8.5.23
Posted by Mark Thomas <ma...@apache.org>.
On 06/04/18 22:10, dharmon wrote:
> environment:
> tomcat 8.5.23
> CentOS 6.9
> jdk8u162 b12
>
> I recently upgraded from tomcat 7.0.61/jdk8u92 to tomcat 8.5.23/jdk8u162.
> Our application war file on tomcat7 has cpu >75% idle per vmstat several
> hours after tomcat start. Same war file on same server with tomcat8.5.23 has
> cpu 0% idle per vmstat several hours after restart. Same server with similar
> request load. Server has 4 CPUs. War file processes https REST web service
> requests.
>
> I used jvisualvm to perform CPU profiling and the method with most self time
> is org.apache.tomcat.util.buf.Utf8Decoder.decodeHasArray. jstack shows there
> are 12 https-jsse-nio-8443-exec-### threads always in runnable state[1].
> They have been in this state for several hours. It appears as if it's stuck
> in infinite loop. Using the thread's nid and lsof, I believe the tcp/https
> connection associated with these threads are no longer
> connected/established.
>
> Any known issues that could be causing this problem? I also tried using the
> NIO2 connector with same issue.
None I am aware of. I wouldn't expect the connector to make any difference.
> I'm on a closed network, thus unable to show jvisualvm screen shots or
> verbose stack traces.
I've taken a good look at the relevant code and nothing immediately
jumps out as a potential cause of this.
It sounds as if this is fairly easy to reproduce so I would suggest
configuring your Tomcat instance to support remote debugging and when it
happens again step through the relevant code to see why /exactly where
it is looping. The aim is to provide us with enough information to
ideally reproduce it but - as a minimum - provide an explanation that
can be used to support a change to fix it.
Mark
>
>
> [1]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.tomcat.util.buf.Utf8Decoder.decodeHasArray(Utf8Decoder.java:159)
> at org.apache.tomcat.util.buf.Utf8Decoder.decodeLoop(Utf8Decoder.java:78)
> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:579)
> at org.apache.tomcat.util.buf.B2CConverter.convert(B2CConverter.java:273)
> at
> org.apache.catalina.connector.InputBuffer.realReadChars(InputBuffer.java:426)
> at
> org.apache.catalina.connector.InputBuffer.checkCharBufferEof(InputBuffer.java:652)
> at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:468)
> at org.apache.catalina.connector.CoyoteReader.read(CoyoteReader.java:101)
> <groovy classes>
> <my application code with is attempting to read http request>
>
>
>
>
>
>
> --
> Sent from: http://tomcat.10.x6.nabble.com/Tomcat-User-f1968778.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org