You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2020/10/22 07:05:30 UTC
[Bug 64839] New: HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Bug ID: 64839
Summary: HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller"
java.lang.NullPointerException
Product: Tomcat 9
Version: 9.0.38
Hardware: HP
OS: Linux
Status: NEW
Severity: blocker
Priority: P2
Component: Catalina
Assignee: dev@tomcat.apache.org
Reporter: arshiya.shariff@ericsson.com
Target Milestone: -----
Sub-Component - Coyote
OS : Redhat Linux
Overview:
Embedded Tomcat version 9.0.38 is implemented to transport http/2 packets
between 2 systems (h2c connection). The number of threads configured in tomcat
is 200, all other values are the tomcat defaults.
We see the below NullPointerException is printed , after which the poller
thread is killed and tomcat is not processing any requests.
Exception in thread "http-nio-x.y.z-1090-ClientPoller"
java.lang.NullPointerException
We did not see any other Exceptions .
We suspect this is because the NullPointerException is not handled in
org.apache.tomat.util.net.NioEndPoint$Poller run() method .
On what scenarios does this occur ?
Can you please help fix this issue.
Build:
Embedded tomcat 9.0.38
Thanks in Advance!!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Remy Maucherat <re...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|UNCONFIRMED |RESOLVED
Resolution|--- |FIXED
--- Comment #17 from Remy Maucherat <re...@apache.org> ---
Please do not reopen the BZ.
No problem has been reported with this code, which never had any try/catch, so
no try/catch needed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Arshiya <ar...@ericsson.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #9 from Arshiya <ar...@ericsson.com> ---
Please find the trace of the NullPointerException:
Exception in thread "http-nio-x.y.x-1090-ClientPoller"
java.lang.NullPointerException
at org.apache.tomcat.util.net.NioEndpoint$Poller.events(NioEndpoint.java:614)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:730)
at java.lang.Thread.run(Thread.java:748)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
Severity|blocker |normal
--- Comment #2 from Mark Thomas <ma...@apache.org> ---
We need the full stack trace to investigate this further.
Note: maxThreads="200" is the Tomcat default. If you are referring to a
different setting please be explicit about which one.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #15 from Mark Thomas <ma...@apache.org> ---
Also back-ported to 8.5.x for 8.5.60 onwards.
As always, the next round of releases will start around the beginning of the
month once all the open issues have been addressed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #1 from Remy Maucherat <re...@apache.org> ---
Can you provide the full stack trace of the exception ?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Arshiya <ar...@ericsson.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #3 from Arshiya <ar...@ericsson.com> ---
This is was the only line that was printed in the logs in production , and we
are not aware why this will happen .
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: [Bug 64839] HTTP2: Exception in thread "http-nio-x.y.z-1090-ClientPoller"
java.lang.NullPointerException
Posted by Martin Grigorov <mg...@apache.org>.
On Thu, Oct 22, 2020 at 5:56 PM <bu...@apache.org> wrote:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
>
> Arshiya <ar...@ericsson.com> changed:
>
> What |Removed |Added
>
> ----------------------------------------------------------------------------
> Status|NEEDINFO |NEW
>
> --- Comment #7 from Arshiya <ar...@ericsson.com> ---
> The BufferOverFlow Exception was printed for about 4 times and then the
> NullPointer Exception.. The exact time stamp of the trace is not known.
>
> Exception in thread "http-nio-x.y.z-1090-exec-20"
> java.nio.BufferOverflowException
>
I think this has been fixed in 9.0.39. Please upgrade!
> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:206)
> at
> org.apache.tomcat.util.net
> .SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
> at
> org.apache.tomcat.util.net
> .SocketWrapperBase.unRead(SocketWrapperBase.java:401)
> at
>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:306)
> at
>
> org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:163)
> at
> org.apache.tomcat.util.net
> .SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
> at
> org.apache.tomcat.util.net
> .NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
>
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:748)
>
> Please let us know if this is of any help ?
>
> Please find the hardware specs /Java version
> Hardware Spec
> Environment - OpenStack Compute hosted on VM
> RAM - RAM - 119478416 - 119 GB
> Cores of CPU - 12
> OS - RHEL 7.4
> Kernel version - 3.10.0-693.58.1.el7.x86_64
> Java Version:1.8.0_241
>
> --
> You are receiving this mail because:
> You are the assignee for the bug.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Arshiya <ar...@ericsson.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #7 from Arshiya <ar...@ericsson.com> ---
The BufferOverFlow Exception was printed for about 4 times and then the
NullPointer Exception.. The exact time stamp of the trace is not known.
Exception in thread "http-nio-x.y.z-1090-exec-20"
java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:206)
at
org.apache.tomcat.util.net.SocketBufferHandler.unReadReadBuffer(SocketBufferHandler.java:100)
at
org.apache.tomcat.util.net.SocketWrapperBase.unRead(SocketWrapperBase.java:401)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:306)
at
org.apache.coyote.http2.Http2AsyncParser$FrameCompletionHandler.completed(Http2AsyncParser.java:163)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1087)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Please let us know if this is of any help ?
Please find the hardware specs /Java version
Hardware Spec
Environment - OpenStack Compute hosted on VM
RAM - RAM - 119478416 - 119 GB
Cores of CPU - 12
OS - RHEL 7.4
Kernel version - 3.10.0-693.58.1.el7.x86_64
Java Version:1.8.0_241
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Remy Maucherat <re...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #5 from Remy Maucherat <re...@apache.org> ---
(In reply to Mark Thomas from comment #4)
> The most that looks
> possible at this stage is to add some additional try/catch blocks with
> logging of exceptions.
I don't see any legitimate place where a NPE can occur, nor any place to add
such a try/catch in a useful way.
Would setting -XX:-OmitStackTraceInFastThrow allow getting the NPE trace ?
If not, then IMO it's a NIO error with the JVM and this should be closed. And
actually, I think it's probably good to check a new JVM (11 or more recent,
lots of fixes and refactorings in NIO).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread "http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Mark Thomas <ma...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |huangzeling8064@dingtalk.co
| |m
--- Comment #18 from Mark Thomas <ma...@apache.org> ---
*** Bug 68908 has been marked as a duplicate of this bug. ***
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #13 from Remy Maucherat <re...@apache.org> ---
A previous refactoring of the Poller.events() could cause this uncaught
exception to occur. This is fixed in 10.0.0-M10 and 9.0.40 where the NPE will
be logged properly and Tomcat should be able to continue processing requests.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Remy Maucherat <re...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #8 from Remy Maucherat <re...@apache.org> ---
You previously reported this BufferOverflowException on the user list, and Mark
found and fixed the (actually totally harmless) issue in Tomcat 9.0.39.
Please provide details on the NPE using the JVM setting.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #14 from Arshiya <ar...@ericsson.com> ---
Thanks a ton Remy!
Any update on the tentative release date of 9.0.40 please .
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #10 from Remy Maucherat <re...@apache.org> ---
Ok, so this NPE will be caught and logged with no major consequences. There is
normally no way it could happen, however (the SocketChannel of the NioChannel
is null, which only happens for the closed channel, which is not supposed to be
in the poller).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #11 from Arshiya <ar...@ericsson.com> ---
Thank you for the swift response Remy.
For a few hours the application accepts requests and processes fine , but
suddenly after this exception is logged , none of the requests are accepted
(tps drops to 0) .
Is this because the external client closes the connection causing this issue in
the Poller ?
If the issue is due to the environment , what can the suspect be on ?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #12 from Remy Maucherat <re...@apache.org> ---
I made a mistake, I was not looking at the right call to events(). So this
should be tightened up [although I don't see how it can end up in this
situation].
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #6 from Mark Thomas <ma...@apache.org> ---
+1 to adding that JVM option and then review tge full stack trace as the next
step.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
--- Comment #4 from Mark Thomas <ma...@apache.org> ---
Without the stack trace there isn't much we can do. The most that looks
possible at this stage is to add some additional try/catch blocks with logging
of exceptions.
It would be helpful to see the following:
- hardware specification
- exact OS version
- exact JRE vendor and version
Given our inability so far to recreate these issues with the provided test
cases and that the issues don't seem to be occurring for other users, I am
beginning to suspect an issue in a component other than Tomcat.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
[Bug 64839] HTTP2: Exception in thread
"http-nio-x.y.z-1090-ClientPoller" java.lang.NullPointerException
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=64839
Venkat <m....@ericsson.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Ever confirmed|1 |0
Status|RESOLVED |UNCONFIRMED
Resolution|FIXED |---
--- Comment #16 from Venkat <m....@ericsson.com> ---
(In reply to Remy Maucherat from comment #13)
> A previous refactoring of the Poller.events() could cause this uncaught
> exception to occur. This is fixed in 10.0.0-M10 and 9.0.40 where the NPE
> will be logged properly and Tomcat should be able to continue processing
> requests.
Iterator<SelectionKey> iterator =
keyCount > 0 ? selector.selectedKeys().iterator() : null;
// Walk through the collection of ready keys and dispatch
// any active event.
while (iterator != null && iterator.hasNext()) {
SelectionKey sk = iterator.next();
iterator.remove();
NioSocketWrapper socketWrapper = (NioSocketWrapper)
sk.attachment();
// Attachment may be null if another thread has called
// cancelledKey()
if (socketWrapper != null) {
processKey(sk, socketWrapper);
}
}
// Process timeouts
timeout(keyCount,hasEvents);
This piece of code in Poller run() is still not in try catch ,any unexpected
exception can cause this thread to die , are you taking care of this as well.
Any hints to reproduce and test this NPE exception in Poller run().
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org