You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kees Jan Koster <kj...@gmail.com> on 2012/05/22 11:11:45 UTC
connection reset errors
Dear Tomcat community,
I am trying to resolve the problem where some client code in Java frequently gets the following error in the logs:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1000)
...
The client code performs a simple HTTP POST request to a Tomcat server (FreeBSD 9.0-STABLE, Java 1.6.0_03, Tomcat 6.0.26). Below is the HTTP connector from server.xml:
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="10000" enableLookups="false" compression="on"
maxThreads="256" bufferSize="9000" />
Tracing the servlet in Tomcat shows that the servlet's doPost() method returns normally and does not show any exceptions (I catch and log Throwable and nothing related is logged). Tracing on an application level shows the posted data to be in the database, as would be for a normal POST. The data is correct.
Of note is that the response time of the post is the same as for a successful post. Under normal circumstances Tomcat processes a post in about 25ms, measured in the client. When I get this exception, the response time is also about that time. I think therefore that I am not timing out anywhere (or the response time would be a lot longer).
First question: It says "Connection reset" and not "Connection reset by peer". What is the difference between the two? I am told that one means a local reset and the other means a remote reset. Where can I find more about this difference?
Second question: how do I analyze such resets? How do I find out who reset the connection and why? How can I replay this in such a way that I can see that?
--
Kees Jan
http://java-monitor.com/
kjkoster@kjkoster.org
+31651838192
Change is good. Granted, it is good in retrospect, but change is good.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear Jose,
>> Yes I am. In finally{} block. Here is the client code:
>
> Calling the disconnect() method of HttpURLConnection may close the
> underlying socket
> if a persistent connection is otherwise idle at that time
> Try don't call it, test it and tell us :-)
>
> http://docs.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html
Ugh. I never realized the socket API was this messy.
At any rate, I call connection.setRequestProperty("Connection", "close"); to indicate that I do not want to make use of persistent connections. You can see that in my client code.
If the underlying socket were closed I would also occasionally see other methods than 'read' get an error. The stack trace I posted is the only one I get, though.
--
Kees Jan
http://java-monitor.com/
kjkoster@kjkoster.org
+31651838192
Change is good. Granted, it is good in retrospect, but change is good.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by Jose María Zaragoza <de...@gmail.com>.
> Yes I am. In finally{} block. Here is the client code:
Calling the disconnect() method of HttpURLConnection may close the
underlying socket
if a persistent connection is otherwise idle at that time
Try don't call it, test it and tell us :-)
http://docs.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html
>
> private Properties push(final Properties request) throws IOException {
> HttpURLConnection connection = null;
> PrintStream out = null;
> InputStream in = null;
> try {
> connection = (HttpURLConnection) pushUrl.openConnection(proxy);
> connection.setRequestMethod("POST");
> connection.setDoOutput(true);
> connection.setConnectTimeout(TWO_MINUTES);
> connection.setReadTimeout(TWO_MINUTES);
> connection.setRequestProperty("Connection", "close");
>
> out = new PrintStream(connection.getOutputStream());
> request.storeToXML(out, null);
> out.flush();
>
> in = connection.getInputStream();
> final Properties response = new Properties();
> response.loadFromXML(in);
>
> return response;
> } finally {
> if (in != null) {
> try {
> in.close();
> } catch (Exception e) {
> // ignore...
> }
> }
> if (out != null) {
> try {
> out.close();
> } catch (Exception e) {
> // ignore...
> }
> }
>
> if (connection != null) {
> try {
> connection.disconnect();
> } catch (Exception e) {
> // ignore...
> }
> }
> }
> }
> }
>
> --
> Kees Jan
>
> http://java-monitor.com/
> kjkoster@kjkoster.org
> +31651838192
>
> Human beings make life so interesting. Do you know that in a universe so full of wonders,
> they have managed to invent boredom. Quite astonishing... -- Terry Pratchett
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear Jose,
> 2012/5/22 Kees Jan Koster <kj...@gmail.com>:
>> Dear Tomcat community,
>>
>> I am trying to resolve the problem where some client code in Java frequently gets the following error in the logs:
>>
>> java.net.SocketException: Connection reset
>> at java.net.SocketInputStream.read(SocketInputStream.java:168)
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
>> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
>> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
>> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1000)
>> ...
>>
>
> Are you calling to disconnect() method ?
Yes I am. In finally{} block. Here is the client code:
private Properties push(final Properties request) throws IOException {
HttpURLConnection connection = null;
PrintStream out = null;
InputStream in = null;
try {
connection = (HttpURLConnection) pushUrl.openConnection(proxy);
connection.setRequestMethod("POST");
connection.setDoOutput(true);
connection.setConnectTimeout(TWO_MINUTES);
connection.setReadTimeout(TWO_MINUTES);
connection.setRequestProperty("Connection", "close");
out = new PrintStream(connection.getOutputStream());
request.storeToXML(out, null);
out.flush();
in = connection.getInputStream();
final Properties response = new Properties();
response.loadFromXML(in);
return response;
} finally {
if (in != null) {
try {
in.close();
} catch (Exception e) {
// ignore...
}
}
if (out != null) {
try {
out.close();
} catch (Exception e) {
// ignore...
}
}
if (connection != null) {
try {
connection.disconnect();
} catch (Exception e) {
// ignore...
}
}
}
}
}
--
Kees Jan
http://java-monitor.com/
kjkoster@kjkoster.org
+31651838192
Human beings make life so interesting. Do you know that in a universe so full of wonders,
they have managed to invent boredom. Quite astonishing... -- Terry Pratchett
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by Jose María Zaragoza <de...@gmail.com>.
2012/5/22 Kees Jan Koster <kj...@gmail.com>:
> Dear Tomcat community,
>
> I am trying to resolve the problem where some client code in Java frequently gets the following error in the logs:
>
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1000)
> ...
>
Are you calling to disconnect() method ?
Regards
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [solved] Re: connection reset errors
Posted by Jose María Zaragoza <de...@gmail.com>.
> The solution was to 1) set acceptCount to a higher value in Tomcat and 2) to configure my OS to allow applications to specify longer accept queues. That last step was the one missing. I had changed acceptCount before, but since the OS was limiting the accept queue length I did not see any improvement.
Thanks for your feedback
One question: modify *only* kern.ipc.somaxconn didn't solve your
problem ? can be any collateral problem with a high value ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
[solved] Re: connection reset errors
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear All,
Well, I managed to track this down. As it turned out, the problem was that I had a rather short TCP listen queue on the Tomcat connector port (100 elements) and that queue was overflowing.
The solution was to 1) set acceptCount to a higher value in Tomcat and 2) to configure my OS to allow applications to specify longer accept queues. That last step was the one missing. I had changed acceptCount before, but since the OS was limiting the accept queue length I did not see any improvement.
More details can be found here:
http://java-monitor.com/forum/showthread.php?t=2492
A big thank you for all that contributed to this thread and helped me understand the problem.
Kees Jan
On 22 May 2012, at 14:45, André Warnier wrote:
> Kees Jan Koster wrote:
>> Dear André,
>>> Assuming that your client is really connecting to that HTTP connector on port 8080 mentioned above..
>> Yes, it has a forwarded port 80 (using FreeBSD ipfw) that also points to 8080, and there is an Apache with mod_proxy_http that hooks into 8081. My tests are on the vanilla port, though.
>
> Can you be a bit clearer on this part ? Do you see the problem happening for 1 in 10 posts, when your client connects directly to Tomcat's HTTP port 8080 ?
> Or is it only when the client connects to Tomcat via either one of these intermediate pieces of machinery ?
>
>>> 1) You are getting a
>>> java.net.SocketException: Connection reset
>>> at java.net.SocketInputStream.read(SocketInputStream.java:168)
>>>
>>> so this appears to happen when/while your java client is reading the response from the server, and it appears to be that the client is expecting to be able to read more data, but finds itself unable to, because the socket has been closed "under his nose".
>> The reading is one area I need to look into: did the client get all data, partial data or none at all. I need to experiment with that.
>>> You say that it happens "frequently", so it's not always.
>> Indeed, not always. About 1 in 10 posts die like this on bad days. Sometimes hours with no issues. No pattern I can discern.
>>> 2) the server itself seems unaware that there is a problem. So it has already written the whole response back to the client, decided it was done with this request, and gone happily to handle other things.
>> Precisely.
>>> That can happen, even if the client has not yet received all data, because between the server and the client there is a lot of piping, and the data may buffered at various levels or still "in transit".
>>>
>>> thus..
>>>
>>> - either the client is misinterpreting the amount of data that it should be reading from the server's response (trying to read more than there actually is)
>>> (on the other hand, I think that the kind of exception you would get in that case would be different, more like "trying to read beyond EOF" or so).
>>> - or something in-between the server and the client closes the connection before all data has been returned to the client (and/or is loosing data).
>>>
>>> It would be helpful to know if this happens when the response is particularly large, or small, or if it is unrelated to the response size.
>> The response is a few bytes. I think it is about 10-20 bytes. Less than a packet, I expect. :)
>
> That is quite strange, I think.
> See below.
>
>
>>> If the server is configured with an AccessLogValve, you should be able to see how big the response was, in bytes. If you have control over the client code, you should be able to add something that logs how many bytes it has read before the exception occurs.
>> What makes the request size interesting? What previous experience are you basing this question on?
>
> Just that intuitively, if a problem happens while reading the response, one would expect that the larger the response is, the more likely that some network issue would show up in the middle.
>
> But now that I say this, going back to your initial message and the stacktrace in it, I see
> ..
> at sun.net.www.http.HttpClient.parseHTTPHeader
> ...
> so the problem seems to show up right away, while the response's HTTP *headers* are being read. So it looks like when the problem happens, the client is not able to read anything at all, not even the headers..
>
> Do all problems show up the same stacktrace, all with a problem while reading/parsing the response headers ?
>
>
>>> Dumping the response HTTP headers to the client logfile would also help finding out what happens. (If the client is an applet running inside of a browser, then a browser add-on would show this easily (like "Live HTTP Headers" for Firefox, or Fiddler2 for IE)).
>> I can check that I see the same problems from a browser using firebug, that is a good idea. Thanks.
>>> Doing a "traceroute" from the client to the server, may also give an idea of what there is actually between the server and the client.
>> mtr reports no packet loss between the two machines I used for testing.
>
> Actually, I was more thinking about some intermediate problematic proxy or something.
> But a traceroute or similar would not show that.
>
> See the first question above, about the direct/indirect connection client-server.
>
>>> And if this all still does not provide any clues, then you're down to a network packet trace, using Wireshark or similar.
>> Packet traces I was hoping to avoid. :(
>
> So far it smells to me like there is some network issue, with some intermediate software or hardware part which is dropping the connection between client and server, after the client has sent the request, but before it even starts receiving the response.
> Is there anything in-between client and server which could have this behaviour, such as when it gets very busy ?
> Do you have any kind of tool which can show you how many requests Tomcat is processing over time, and if these problems happen when it is handling lots of requests ?
> (Not that the problem appears to be at the Tomcat level, but just to check how busy the network may be at such times)
>
> Another thing : your client is effectively requesting non-keepalive connections, so Tomcat will close the connection after sending the response to each request. And your clients have to rebuild a new connection for each request.
> If the same client(s) make lots of small requests one after another, this may be counter-productive, because each connection build-up requires several packets going back and forth. Also, on the server side, when a connection is being closed, it will nevertheless "linger" for a while in CLOSE_WAIT state, waiting for the client's TCP stack to acknowledge the CLOSE. I have seen cases where a large number of such connections being in CLOSE_WAIT triggered bizarre issues, such as a server becoming unable to accept new TCP connections for a while.
> It may be worth checking how many of such CLOSE_WAIT connections you have over time, and if this relates to when the problems happen.
> netstat -pan | grep CLOSE_WAIT
> would show this. If more than a couple of hundreds show up, I'd become suspicious of something like that.
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Kees Jan
http://java-monitor.com/
kjkoster@kjkoster.org
+31651838192
The secret of success lies in the stability of the goal. -- Benjamin Disraeli
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear André,
Took me a while to answer, because I wanted to get more precise readings.
>>> Assuming that your client is really connecting to that HTTP connector on port 8080 mentioned above..
>> Yes, it has a forwarded port 80 (using FreeBSD ipfw) that also points to 8080, and there is an Apache with mod_proxy_http that hooks into 8081. My tests are on the vanilla port, though.
>
> Can you be a bit clearer on this part ? Do you see the problem happening for 1 in 10 posts, when your client connects directly to Tomcat's HTTP port 8080 ?
> Or is it only when the client connects to Tomcat via either one of these intermediate pieces of machinery ?
I re-ran my tests, making sure that I was positive about the path the packets travel. Here is what I fished out of the log of a client app running on the same machine as the server:
java.net.SocketException: Connection reset by peer
at java.net.PlainSocketImpl.socketSetOption(Native Method)
at java.net.AbstractPlainSocketImpl.setOption(AbstractPlainSocketImpl.java:267)
at java.net.Socket.setTcpNoDelay(Socket.java:940)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:400)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:483)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:213)
at sun.net.www.http.HttpClient.New(HttpClient.java:300)
at sun.net.www.http.HttpClient.New(HttpClient.java:316)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:992)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:971)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:846)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1087)
....
This particular client connects on the same machine as the Tomcat server, using port 8080 (the configured HTTP connector for this Tomcat server). The URL used is http://localhost:8080/<webapp>.
So whatever is happening, it is not happening on the network or in the ISP's hardware. It seems to be local to my machine and with this I feel that Tomcat is a suspect again. This goes against some of the things I said earlier. I was wrong, sorry.
Is there any way to overwhelm a Tomcat connector without it serving 503 responses? This particular connector is maxed out at 256 connections, with about 5% actually busy at any given time (as reported by JMX), with spikes of up to 50% busy every hour or so.
> Another thing : your client is effectively requesting non-keepalive connections, so Tomcat will close the connection after sending the response to each request. And your clients have to rebuild a new connection for each request.
> If the same client(s) make lots of small requests one after another, this may be counter-productive, because each connection build-up requires several packets going back and forth. Also, on the server side, when a connection is being closed, it will nevertheless "linger" for a while in CLOSE_WAIT state, waiting for the client's TCP stack to acknowledge the CLOSE. I have seen cases where a large number of such connections being in CLOSE_WAIT triggered bizarre issues, such as a server becoming unable to accept new TCP connections for a while.
I know this, but since I know the traffic pattern (connections are used once per minute) I opted to make the connections non-keep-alive from the client side. That way, the connections that come from browsers can make use of keep-alive for performance.
I guess I should have gone for a model where I have a non-keel-alive HTTP connector on Tomcat for the clients and another keep-alive-enabled connector for the browser traffic. Hmmm. It may not be too late for that, but first I'd like some proof this is an issue. :)
> It may be worth checking how many of such CLOSE_WAIT connections you have over time, and if this relates to when the problems happen.
> netstat -pan | grep CLOSE_WAIT
> would show this. If more than a couple of hundreds show up, I'd become suspicious of something like that.
I now graph these in Munin and I see spikes of up to 100 sockets in that state. You say hundreds are a problem, what about one hundred?
--
Kees Jan
http://java-monitor.com/
kjkoster@kjkoster.org
+31651838192
Repairing cannot be completed, you can only stop doing it.
-- Belarusian proverb
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by André Warnier <aw...@ice-sa.com>.
Kees Jan Koster wrote:
> Dear André,
>
>> Assuming that your client is really connecting to that HTTP connector on port 8080 mentioned above..
>
> Yes, it has a forwarded port 80 (using FreeBSD ipfw) that also points to 8080, and there is an Apache with mod_proxy_http that hooks into 8081. My tests are on the vanilla port, though.
>
Can you be a bit clearer on this part ? Do you see the problem happening for 1 in 10
posts, when your client connects directly to Tomcat's HTTP port 8080 ?
Or is it only when the client connects to Tomcat via either one of these intermediate
pieces of machinery ?
>> 1) You are getting a
>> java.net.SocketException: Connection reset
>> at java.net.SocketInputStream.read(SocketInputStream.java:168)
>>
>> so this appears to happen when/while your java client is reading the response from the server, and it appears to be that the client is expecting to be able to read more data, but finds itself unable to, because the socket has been closed "under his nose".
>
> The reading is one area I need to look into: did the client get all data, partial data or none at all. I need to experiment with that.
>
>> You say that it happens "frequently", so it's not always.
>
> Indeed, not always. About 1 in 10 posts die like this on bad days. Sometimes hours with no issues. No pattern I can discern.
>
>> 2) the server itself seems unaware that there is a problem. So it has already written the whole response back to the client, decided it was done with this request, and gone happily to handle other things.
>
> Precisely.
>
>> That can happen, even if the client has not yet received all data, because between the server and the client there is a lot of piping, and the data may buffered at various levels or still "in transit".
>>
>> thus..
>>
>> - either the client is misinterpreting the amount of data that it should be reading from the server's response (trying to read more than there actually is)
>> (on the other hand, I think that the kind of exception you would get in that case would be different, more like "trying to read beyond EOF" or so).
>> - or something in-between the server and the client closes the connection before all data has been returned to the client (and/or is loosing data).
>>
>> It would be helpful to know if this happens when the response is particularly large, or small, or if it is unrelated to the response size.
>
> The response is a few bytes. I think it is about 10-20 bytes. Less than a packet, I expect. :)
That is quite strange, I think.
See below.
>
>> If the server is configured with an AccessLogValve, you should be able to see how big the response was, in bytes. If you have control over the client code, you should be able to add something that logs how many bytes it has read before the exception occurs.
>
> What makes the request size interesting? What previous experience are you basing this question on?
Just that intuitively, if a problem happens while reading the response, one would expect
that the larger the response is, the more likely that some network issue would show up in
the middle.
But now that I say this, going back to your initial message and the stacktrace in it, I see
..
at sun.net.www.http.HttpClient.parseHTTPHeader
...
so the problem seems to show up right away, while the response's HTTP *headers* are being
read. So it looks like when the problem happens, the client is not able to read anything
at all, not even the headers..
Do all problems show up the same stacktrace, all with a problem while reading/parsing the
response headers ?
>
>> Dumping the response HTTP headers to the client logfile would also help finding out what happens. (If the client is an applet running inside of a browser, then a browser add-on would show this easily (like "Live HTTP Headers" for Firefox, or Fiddler2 for IE)).
>
> I can check that I see the same problems from a browser using firebug, that is a good idea. Thanks.
>
>> Doing a "traceroute" from the client to the server, may also give an idea of what there is actually between the server and the client.
>
> mtr reports no packet loss between the two machines I used for testing.
Actually, I was more thinking about some intermediate problematic proxy or something.
But a traceroute or similar would not show that.
See the first question above, about the direct/indirect connection client-server.
>
>> And if this all still does not provide any clues, then you're down to a network packet trace, using Wireshark or similar.
>
>
> Packet traces I was hoping to avoid. :(
So far it smells to me like there is some network issue, with some intermediate software
or hardware part which is dropping the connection between client and server, after the
client has sent the request, but before it even starts receiving the response.
Is there anything in-between client and server which could have this behaviour, such as
when it gets very busy ?
Do you have any kind of tool which can show you how many requests Tomcat is processing
over time, and if these problems happen when it is handling lots of requests ?
(Not that the problem appears to be at the Tomcat level, but just to check how busy the
network may be at such times)
Another thing : your client is effectively requesting non-keepalive connections, so Tomcat
will close the connection after sending the response to each request. And your clients
have to rebuild a new connection for each request.
If the same client(s) make lots of small requests one after another, this may be
counter-productive, because each connection build-up requires several packets going back
and forth. Also, on the server side, when a connection is being closed, it will
nevertheless "linger" for a while in CLOSE_WAIT state, waiting for the client's TCP stack
to acknowledge the CLOSE. I have seen cases where a large number of such connections
being in CLOSE_WAIT triggered bizarre issues, such as a server becoming unable to accept
new TCP connections for a while.
It may be worth checking how many of such CLOSE_WAIT connections you have over time, and
if this relates to when the problems happen.
netstat -pan | grep CLOSE_WAIT
would show this. If more than a couple of hundreds show up, I'd become suspicious of
something like that.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by Kees Jan Koster <kj...@gmail.com>.
Dear André,
> Assuming that your client is really connecting to that HTTP connector on port 8080 mentioned above..
Yes, it has a forwarded port 80 (using FreeBSD ipfw) that also points to 8080, and there is an Apache with mod_proxy_http that hooks into 8081. My tests are on the vanilla port, though.
> 1) You are getting a
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
>
> so this appears to happen when/while your java client is reading the response from the server, and it appears to be that the client is expecting to be able to read more data, but finds itself unable to, because the socket has been closed "under his nose".
The reading is one area I need to look into: did the client get all data, partial data or none at all. I need to experiment with that.
> You say that it happens "frequently", so it's not always.
Indeed, not always. About 1 in 10 posts die like this on bad days. Sometimes hours with no issues. No pattern I can discern.
> 2) the server itself seems unaware that there is a problem. So it has already written the whole response back to the client, decided it was done with this request, and gone happily to handle other things.
Precisely.
> That can happen, even if the client has not yet received all data, because between the server and the client there is a lot of piping, and the data may buffered at various levels or still "in transit".
>
> thus..
>
> - either the client is misinterpreting the amount of data that it should be reading from the server's response (trying to read more than there actually is)
> (on the other hand, I think that the kind of exception you would get in that case would be different, more like "trying to read beyond EOF" or so).
> - or something in-between the server and the client closes the connection before all data has been returned to the client (and/or is loosing data).
>
> It would be helpful to know if this happens when the response is particularly large, or small, or if it is unrelated to the response size.
The response is a few bytes. I think it is about 10-20 bytes. Less than a packet, I expect. :)
> If the server is configured with an AccessLogValve, you should be able to see how big the response was, in bytes. If you have control over the client code, you should be able to add something that logs how many bytes it has read before the exception occurs.
What makes the request size interesting? What previous experience are you basing this question on?
> Dumping the response HTTP headers to the client logfile would also help finding out what happens. (If the client is an applet running inside of a browser, then a browser add-on would show this easily (like "Live HTTP Headers" for Firefox, or Fiddler2 for IE)).
I can check that I see the same problems from a browser using firebug, that is a good idea. Thanks.
> Doing a "traceroute" from the client to the server, may also give an idea of what there is actually between the server and the client.
mtr reports no packet loss between the two machines I used for testing.
> And if this all still does not provide any clues, then you're down to a network packet trace, using Wireshark or similar.
Packet traces I was hoping to avoid. :(
--
Kees Jan
http://java-monitor.com/
kjkoster@kjkoster.org
+31651838192
The secret of success lies in the stability of the goal. -- Benjamin Disraeli
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: connection reset errors
Posted by André Warnier <aw...@ice-sa.com>.
Kees Jan Koster wrote:
> Dear Tomcat community,
>
> I am trying to resolve the problem where some client code in Java frequently gets the following error in the logs:
>
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:652)
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1000)
> ...
>
> The client code performs a simple HTTP POST request to a Tomcat server (FreeBSD 9.0-STABLE, Java 1.6.0_03, Tomcat 6.0.26). Below is the HTTP connector from server.xml:
>
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="10000" enableLookups="false" compression="on"
> maxThreads="256" bufferSize="9000" />
>
> Tracing the servlet in Tomcat shows that the servlet's doPost() method returns normally and does not show any exceptions (I catch and log Throwable and nothing related is logged). Tracing on an application level shows the posted data to be in the database, as would be for a normal POST. The data is correct.
>
> Of note is that the response time of the post is the same as for a successful post. Under normal circumstances Tomcat processes a post in about 25ms, measured in the client. When I get this exception, the response time is also about that time. I think therefore that I am not timing out anywhere (or the response time would be a lot longer).
>
> First question: It says "Connection reset" and not "Connection reset by peer". What is the difference between the two? I am told that one means a local reset and the other means a remote reset. Where can I find more about this difference?
>
> Second question: how do I analyze such resets? How do I find out who reset the connection and why? How can I replay this in such a way that I can see that?
> --
Hi.
Assuming that your client is really connecting to that HTTP connector on port 8080
mentioned above..
1) You are getting a
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
so this appears to happen when/while your java client is reading the response from the
server, and it appears to be that the client is expecting to be able to read more data,
but finds itself unable to, because the socket has been closed "under his nose".
You say that it happens "frequently", so it's not always.
2) the server itself seems unaware that there is a problem. So it has already written the
whole response back to the client, decided it was done with this request, and gone happily
to handle other things.
That can happen, even if the client has not yet received all data, because between the
server and the client there is a lot of piping, and the data may buffered at various
levels or still "in transit".
thus..
- either the client is misinterpreting the amount of data that it should be reading from
the server's response (trying to read more than there actually is)
(on the other hand, I think that the kind of exception you would get in that case would be
different, more like "trying to read beyond EOF" or so).
- or something in-between the server and the client closes the connection before all data
has been returned to the client (and/or is loosing data).
It would be helpful to know if this happens when the response is particularly large, or
small, or if it is unrelated to the response size.
If the server is configured with an AccessLogValve, you should be able to see how big the
response was, in bytes. If you have control over the client code, you should be able to
add something that logs how many bytes it has read before the exception occurs.
Dumping the response HTTP headers to the client logfile would also help finding out what
happens. (If the client is an applet running inside of a browser, then a browser add-on
would show this easily (like "Live HTTP Headers" for Firefox, or Fiddler2 for IE)).
Doing a "traceroute" from the client to the server, may also give an idea of what there
is actually between the server and the client.
And if this all still does not provide any clues, then you're down to a network packet
trace, using Wireshark or similar.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org