You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Patkar Omkar Anant <om...@gi-de.com> on 2023/02/15 12:24:42 UTC
Tomcat Issue = org.apache.tomcat.util.net.Acceptor.run Socket accept failed. java.io.IOException: Duplicate accept detected.
Hi All,
We are facing an issue with Apache Tomcat 9. I had posted this issue in Bugzilla forum, and from there I was re-directed here.
The link to the bug is = https://bz.apache.org/bugzilla/show_bug.cgi?id=66476
I will elaborate the details here: -
----------------------------------
ERROR: -
----------------------------------
10-Feb-2023 02:16:19.618 SEVERE [http-nio-80-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
java.io.IOException: Duplicate accept detected. This is a known OS bug. Please consider reporting that you are affected: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1924298
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:548)
at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:78)
at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:129)
at java.lang.Thread.run(Thread.java:748)
----------------------------------
----------------------------------
Env details
----------------------------------
Apache Tomcat version : - Apache Tomcat/9.0.58
Frequency of occurrence : - Sometimes
Reproducible consistently ? : - No, it appears sometimes ....randomly...and on tomcat restart it goes away but it may appear randomly sometime later
OS = CentOS
Kernel (uname -sr) = Linux 6.0.9-1.el7.elrepo.x86_64
Server arch = amd64
----------------------------------
----------------------------------
Setup: -
----------------------------------
We are using Camunda 7.16 BPMN tool which comes with inbuilt Apache Tomcat 9.0.58. This is deployed on our Linux VM (CentOS). The VM is within our company network, and ... via company's firewall another trusted application on Azure cloud (outside company network)...is only allowed to make REST API calls to Camunda (hosted on Tomcat). When Tomcat is started all works fine, requests placed from trusted application on Azure Cloud are able to reach our Tomcat and our application works on it, until out of the blue and randomly above error occurs and tomcat no longer serves request coming from external application on cloud.
On the other hand, if we fire similar request from within our company network machines as many times as we want... as frequently as possible, in such scenario, the above error does not occur and Tomcat allows all internal requests.
In other words, it seems above issue occurs only for external requests and not internal ones.
What has been tried so far ?: -
From several posts after google search it was found that Linux Kernel version could help. Previously it was 3.x and now it is upgraded to 6.0.9-1 and Apache Tomcat version is 9.0.58... still we face the above error.
----------------------------------
----------------------------------
Queries: -
----------------------------------
1) Why it behaves strangely with external requests and not with internal ones ?
2) How can we consistently reproduce the issue ?
3) What could be the possible cause and how to fix it ?
----------------------------------
Any help in this regard can be very helpful. On Bugzilla, Mark has recommended to upgrade to Apache Tomcat 9.0.59 or higher. But we want to be sure that is the fix, because the error that we are getting is occurring only randomly and that too for external requests and not internal requests.
Let me know if you need any more information.
Thanks and Regards
Omkar Patkar
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Issue = org.apache.tomcat.util.net.Acceptor.run Socket accept failed. java.io.IOException: Duplicate accept detected.
Posted by Mark Thomas <ma...@apache.org>.
On 17/02/2023 07:53, Patkar Omkar Anant wrote:
> Hi Mark Thomas,
>
> Few queries from my side.
>
> 1) You have indicated it could be false positive ? But how do we confirm that ? Also, Its not just a warning, the requests are not processed any more.
The bug referenced in the error message affects particular versions of
the Linux kernel. The bug was fixed in Linux kernel 5.10-rc6. You have
updated to 6.0.9 so you will no longer be seeing the bug.
The way Tomcat detects the bug in 9.0.58 is susceptible to false
positives. See bug 65776 for details. If you continue to see the error
message with a fixed Linux kernel then the most likely explanation is
that you are seeing false positives.
> 2) "To be clear, no further requests can be served from the external application but internal clients can continue?" ---> Yes. For requests coming from internal clients there is no issue, Tomcat allows to process such requests. But, if it stops randomly for requests coming from external client, it will continue to stop unless tomcat is restarted.
That is interesting but we need more information to figure out what it
means.
> 3) "you need to upgrade as the detection got a lot better in 9.0.59 onwards." ----> does it mean, it will output more information or the issue will get solved ?
It means that the issue with false positives (bug 65776) has been
resolved so if you see this error message with Tomcat 9.0.59 onwards it
is likely you are affected by the Linux bug.
> 4) " Do internal and external requests go to the same connector?" ----> I think yes.... but to confirm, let me know steps I shall follow them to confirm.
For internal and external requests to go to a different connector, there
will need to be at least two Connectors defined in server.xml. Please
provide all the Connector elements from server.xml, replacing any
passwords etc with xxx.
For the internal and external clients to be using the same Connector
they must be connecting to the same host and port (but depending on
Connector and configuration it may be possible for them to be using
different IP addresses). Please provide the host name(s) and port(s) the
internal and external clients are connecting to.
The firewall may also be relevant. What OSI layer is it operating at? Is
a reverse proxy in use at all?
Once the external clients start to fail, do you see a duplicate accept
error message in the logs for every attempt the external clients make?
Or is there just one error message and all subsequent external requests
fail without further messages in the logs?
When the external requests are failing, capture a thread dump and
provide that.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat Issue = org.apache.tomcat.util.net.Acceptor.run Socket accept failed. java.io.IOException: Duplicate accept detected.
Posted by Patkar Omkar Anant <om...@gi-de.com>.
Hi Mark Thomas,
Few queries from my side.
1) You have indicated it could be false positive ? But how do we confirm that ? Also, Its not just a warning, the requests are not processed any more.
2) "To be clear, no further requests can be served from the external application but internal clients can continue?" ---> Yes. For requests coming from internal clients there is no issue, Tomcat allows to process such requests. But, if it stops randomly for requests coming from external client, it will continue to stop unless tomcat is restarted.
3) "you need to upgrade as the detection got a lot better in 9.0.59 onwards." ----> does it mean, it will output more information or the issue will get solved ?
4) " Do internal and external requests go to the same connector?" ----> I think yes.... but to confirm, let me know steps I shall follow them to confirm.
-----Original Message-----
From: Mark Thomas <ma...@apache.org>
Sent: 16 February 2023 01:13
To: users@tomcat.apache.org
Subject: Re: Tomcat Issue = org.apache.tomcat.util.net.Acceptor.run Socket accept failed. java.io.IOException: Duplicate accept detected.
On 15/02/2023 12:24, Patkar Omkar Anant wrote:
> Hi All,
>
> We are facing an issue with Apache Tomcat 9. I had posted this issue in Bugzilla forum, and from there I was re-directed here.
>
> The link to the bug is =
> https://bz.apache.org/bugzilla/show_bug.cgi?id=66476
>
> I will elaborate the details here: -
>
> ----------------------------------
> ERROR: -
> ----------------------------------
> 10-Feb-2023 02:16:19.618 SEVERE [http-nio-80-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
> java.io.IOException: Duplicate accept detected. This is a known OS bug. Please consider reporting that you are affected:
The only result of that error message is that the duplicate accept is ignored. It should be transparent to the client as request processing will continue on the original connection.
If you see this message as a result of a false positive then the client connection will be dropped. That is obviously more problematic.
> ----------------------------------
> We are using Camunda 7.16 BPMN tool which comes with inbuilt Apache Tomcat 9.0.58. This is deployed on our Linux VM (CentOS). The VM is within our company network, and ... via company's firewall another trusted application on Azure cloud (outside company network)...is only allowed to make REST API calls to Camunda (hosted on Tomcat). When Tomcat is started all works fine, requests placed from trusted application on Azure Cloud are able to reach our Tomcat and our application works on it, until out of the blue and randomly above error occurs and tomcat no longer serves request coming from external application on cloud.
To be clear, no further requests can be served from the external application but internal clients can continue?
> What has been tried so far ?: -
> From several posts after google search it was found that Linux Kernel version could help. Previously it was 3.x and now it is upgraded to 6.0.9-1 and Apache Tomcat version is 9.0.58... still we face the above error.
That suggests you are seeing false positives rather than the Linux bug.
To avoid the false positives, you need to upgrade as the detection got a lot better in 9.0.59 onwards.
> 1) Why it behaves strangely with external requests and not with internal ones ?
Do internal and external requests go to the same connector?
It would be worth running Wireshark or equivalent to look at the external traffic that is being received. In particular, look at the source port for the requests once the problem starts.
> 2) How can we consistently reproduce the issue ?
If you want to test the original Linux bug, steps to reproduce are in the Ubuntu issue referenced in the error message.
If you want to recreate the issue you are seeing (which I suspect is specific to your environment) you'll need to figure out what is happening first.
> 3) What could be the possible cause and how to fix it ?
Assuming you are seeing false positives with the Linux bug detection, the client using the same port for multiple requests. If the Linux bug error message is a red herring, then an Acceptor crash might explain what you are seeing.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat Issue = org.apache.tomcat.util.net.Acceptor.run Socket accept failed. java.io.IOException: Duplicate accept detected.
Posted by Mark Thomas <ma...@apache.org>.
On 15/02/2023 12:24, Patkar Omkar Anant wrote:
> Hi All,
>
> We are facing an issue with Apache Tomcat 9. I had posted this issue in Bugzilla forum, and from there I was re-directed here.
>
> The link to the bug is = https://bz.apache.org/bugzilla/show_bug.cgi?id=66476
>
> I will elaborate the details here: -
>
> ----------------------------------
> ERROR: -
> ----------------------------------
> 10-Feb-2023 02:16:19.618 SEVERE [http-nio-80-Acceptor] org.apache.tomcat.util.net.Acceptor.run Socket accept failed
> java.io.IOException: Duplicate accept detected. This is a known OS bug. Please consider reporting that you are affected:
The only result of that error message is that the duplicate accept is
ignored. It should be transparent to the client as request processing
will continue on the original connection.
If you see this message as a result of a false positive then the client
connection will be dropped. That is obviously more problematic.
> ----------------------------------
> We are using Camunda 7.16 BPMN tool which comes with inbuilt Apache Tomcat 9.0.58. This is deployed on our Linux VM (CentOS). The VM is within our company network, and ... via company's firewall another trusted application on Azure cloud (outside company network)...is only allowed to make REST API calls to Camunda (hosted on Tomcat). When Tomcat is started all works fine, requests placed from trusted application on Azure Cloud are able to reach our Tomcat and our application works on it, until out of the blue and randomly above error occurs and tomcat no longer serves request coming from external application on cloud.
To be clear, no further requests can be served from the external
application but internal clients can continue?
> What has been tried so far ?: -
> From several posts after google search it was found that Linux Kernel version could help. Previously it was 3.x and now it is upgraded to 6.0.9-1 and Apache Tomcat version is 9.0.58... still we face the above error.
That suggests you are seeing false positives rather than the Linux bug.
To avoid the false positives, you need to upgrade as the detection got a
lot better in 9.0.59 onwards.
> 1) Why it behaves strangely with external requests and not with internal ones ?
Do internal and external requests go to the same connector?
It would be worth running Wireshark or equivalent to look at the
external traffic that is being received. In particular, look at the
source port for the requests once the problem starts.
> 2) How can we consistently reproduce the issue ?
If you want to test the original Linux bug, steps to reproduce are in
the Ubuntu issue referenced in the error message.
If you want to recreate the issue you are seeing (which I suspect is
specific to your environment) you'll need to figure out what is
happening first.
> 3) What could be the possible cause and how to fix it ?
Assuming you are seeing false positives with the Linux bug detection,
the client using the same port for multiple requests. If the Linux bug
error message is a red herring, then an Acceptor crash might explain
what you are seeing.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org