You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Mark Claassen <ma...@donnell.com> on 2013/01/16 21:06:32 UTC
Connection Reset errors
I have a user getting a lot of Connection Reset errors. I did not think this had do to with HttpClient, so much as other factors on
her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient
before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
This user is fine for a while, and then will get the Connection Reset error. She then just retries the request, and it works. Any
tips on how to resolve this would be greatly appreciated.
Thanks,
Mark
We are using Tomcat 7.0.27, and the Connector is configured like this:
<Connector
port="5502"
protocol="org.apache.coyote.http11.Http11NioProtocol"
server="Unknown"
connectionLinger="0"
socket.soTimeout="300000"
SSLEnabled="true"
maxThreads="150"
scheme="https"
secure="true"
clientAuth="false"
keystoreFile="-----"
keystoreType="PKCS12"
keystorePass="-----"
sslProtocol="TLS"
ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
---- (1) ---- Throwable - Class (class java.net.SocketException)
Message (Connection reset)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2013-01-18 at 13:47 -0500, Mark Claassen wrote:
> Oh, I forgot to add that all my requests are POST requests.
>
Hi Mark
I wish I could help you but I just do not know how. All I can tell you
is that connection resets can happen for all sorts of reasons. There is
no one common cause.
Writing out request message _may_ appear successful simply because the
whole message was small enough to fit into the socket buffer. It does
not necessarily mean it got transferred to the opposite endpoint.
The problem with POST messages is that usually they are not idempotent.
This is precisely the reason why HttpClient does not retry entity
closing methods automatically. HttpURLConnection used to be careless
enough to silently retry POST requests several times prior to throwing
an I/O exception. I hope it is no longer the case. If it is, then
HttpURLConnection is broken beyond redemption.
Oleg
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
>
> However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
>
> Mark
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
>
> > It may well be that HttpURLConnection silently retries failed requests
> Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
>
> She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
>
> public boolean isStale() {
> if (!isOpen()) {
> return true;
> }
> if (isEof()) {
> return true;
> }
> try {
> this.inbuffer.isDataAvailable(1);
> return isEof();
> } catch (SocketTimeoutException ex) {
> return false;
> } catch (IOException ex) {
> return true;
> }
> }
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> >
>
> It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
>
>
> > Some other things of note:
> > - She is using JRE 6 Update 13. (There is a long story behind this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> > ier);
> >
>
> I do not thing any of these should matter.
>
> Oleg
>
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and speed
> > > does not seem to be a factory (especially with 4.2.3)
> > >
> > > I got in configured and distributed to this user and she is still
> > > having the same problem. Any other ideas? I double checked my
> > > code, and it looks like everything is correct. (She is not using a
> > > proxy.)
> > >
> >
> > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> >
> > Oleg
> >
> >
> > >
> > > Method that makes the request (from multiple threads)
> > > {
> > > response = httpClient.execute(request);
> > > }
> > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > {
> > > HttpParams params = new BasicHttpParams();
> > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > configureProxy(params,protocol);
> > >
> > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > }
> > > private void configureProxy(HttpParams params, String scheme) {
> > > Proxy proxy = getProxy();
> > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > //SSL or not, we set the proxy up as HTTP
> > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > if (!scheme.equals(SCHEME_HTTP)) {
> > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > }
> > > }
> > >
> > > }
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > >
> > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > >
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > > Boolean.TRUE);
> > > >
> > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > >
> > > > Mark
> > > >
> > >
> > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > >
> > > Oleg
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > >
> > > > I have a user getting a lot of Connection Reset errors. I did not
> > > > think this had do to with HttpClient, so much as other factors on
> > > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > >
> > > > This user is fine for a while, and then will get the Connection
> > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > <Connector
> > > > port="5502"
> > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > server="Unknown"
> > > > connectionLinger="0"
> > > > socket.soTimeout="300000"
> > > > SSLEnabled="true"
> > > > maxThreads="150"
> > > > scheme="https"
> > > > secure="true"
> > > > clientAuth="false"
> > > > keystoreFile="-----"
> > > > keystoreType="PKCS12"
> > > > keystorePass="-----"
> > > > sslProtocol="TLS"
> > > >
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > >
> > > >
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > Message (Connection reset)
> > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Looks like I start my counting at 1, so for me it dies always on the 100th connection. I put this on another laptop, (mine has Vista, his is Windows 7) and it dies too, but not at the same place. His dies on attempt:
2100
300
1300
1200
2500
See a pattern: always a multiple of 100. What is going on?!!
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 2:59 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
CLUE!
I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
I about doubled the size of the message I send, and it makes no difference.
I put the iteration count in the HttpHeader so I could track it in the Tomcat access log. Tomcat shows that it receives this message (message 100) and that it thinks it succeeded (status == 200)
The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 2:01 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
FINE: << " </BODY>[\n]"
Thread-00, setSoTimeout(45000) called
[snip]
Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]: length = 176
[snip]
[Raw read]: length = 5
0000: 17 03 01 03 0A .....
Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
0010: C8 08 ..
Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception sending alert: java.net.SocketException: Connection reset by peer: socket write error Thread-00, called closeSocket() Thread-00, called close() Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 12:39 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Don't know if this will help either...but here is hoping!
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 9:03 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> I ran my test program using 4.01 and it seems to fail even faster.
> (Switching a few things around to get it to compile, or course. Using
> ThreadSafeClientConnManager.)
>
> This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
>
> Still, the Java.net stuff is completely error free. (I can't say for
> sure if she would notice automatic retries, if they occurred.)
>
> Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
>
Mark,
This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
So, running your application with SSL debugging turned on should be more
informative:
http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
Oleg
> ------------------------------------
> Stack trace and debug using 4.0.1
>
> FINE: Stale connection check
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013 12:08:45
> PM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Closing the connection.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection shut down
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> releaseConnection
> FINE: Released connection is not reusable.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> FINE: Releasing connection
> [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> FINE: Notifying no-one, there are no waiting threads
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> at java.lang.Thread.run(Thread.java:662)
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 12:30 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
>
> The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
>
> First call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> HttpRequestExecutor.closeConnection:144
> HttpRequestExecutor.execute:131
> DefaultRequestDirector.tryExecute:717
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Second call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> DefaultRequestDirector.tryExecute:723
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Third call to close()
> ----------------------
> DefaultClientConnection.close:168
> HttpPoolEntry.close:89
> AbstractConnPool.release:323
> PoolingClientConnectionManager.releaseConnection:278
> ManagedClientConnectionImpl.abortConnection:463
> DefaultRequestDirector.abortConnection:1157
> DefaultRequestDirector.execute:621
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:41 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
>
> Below is the last part of what I got.
>
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> closeExpiredConnections
> FINE: Closing expired connections
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> requestConnection
> FINE: Connection request: [route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> leaseConnection
> FINE: Connection leased: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Stale connection check
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAuthCache process
> FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> 11:38:01 AM
> org.apache.http.client.protocol.RequestTargetAuthentication process
> FINE: Target auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestProxyAuthentication process
> FINE: Proxy auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013 11:38:01
> AM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Closing the connection.
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 0 of 1; total allocated: 0 of 1]
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> at java.lang.Thread.run(Thread.java:662)
> Java Result: 1
> BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:25 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:07 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Before the upgrade, she was using HttpClient 4.01.
> ---
> Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
>
> The website it is connecting to is just a Tomcat server running the
> default ROOT webapp. It is delivering the default HTML page we have
> set up on that. I would imagine that it is ignoring the POST and
> treating it like a GET
>
> In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 25, 2013 7:46 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> Mark
>
> You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
>
> There are probably several things you might want to try out
>
> (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
>
> (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
>
> (3) Otherwise there is no way around getting very close and personal with Wireshark.
>
> Oleg
>
>
>
> On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> >
> > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Thursday, January 24, 2013 10:29 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> >
> > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> >
> > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > --
> >
> > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > I am trying to force the issue doing a lot of requests all at once.
> > I reduced my MAX down to 1 to force contention. Everything seems to
> > work pretty well. However, I did get the Connection Reset error
> > twice on my machine! The logs were fairly regular looking since
> > there is only one connection at a time. However, when this
> > happened, it looks like it is sending 2 request at once. This
> > shouldn't be happening because there is only one connection.
> > (Granted, it is hard to tell if the logging is indicating a
> > threading issue, or if the "Connection Reset" error caused the
> > logging abnormality.)
> >
> > I don't need to block around the "httpClient.execute(request);" do I?
> >
> > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > --
> >
> > I do this to establish the connection:
> >
> > This is run once in the setup
> > -----------------------------
> > connectionManager = new
> > PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> >
> >
> >
> > This is called when I want to make a request
> > --------------------------------------------
> > response = httpClient.execute(request);
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 5:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Well, the user went all day without any errors while using the Java.net connector.
> >
> > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> >
> > Mark
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:47 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Oh, I forgot to add that all my requests are POST requests.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:02 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> >
> > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> >
> > Mark
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 12:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> >
> > > It may well be that HttpURLConnection silently retries failed
> > > requests
> > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> >
> > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> >
> > public boolean isStale() {
> > if (!isOpen()) {
> > return true;
> > }
> > if (isEof()) {
> > return true;
> > }
> > try {
> > this.inbuffer.isDataAvailable(1);
> > return isEof();
> > } catch (SocketTimeoutException ex) {
> > return false;
> > } catch (IOException ex) {
> > return true;
> > }
> > }
> >
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 11:24 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > >
> >
> > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> >
> >
> > > Some other things of note:
> > > - She is using JRE 6 Update 13. (There is a long story behind
> > > this.)
> > > - The application is launched via Webstart
> > > - The SocketFactory is created with: fact = new
> > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),v
> > > er
> > > if
> > > ier);
> > >
> >
> > I do not thing any of these should matter.
> >
> > Oleg
> >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 6:17 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > Thanks for the explanation.
> > > > > pro-actively evicting expired connections and connections that
> > > > > have been idle
> > > > Seems like just doing the stale check would more reliable, and
> > > > speed does not seem to be a factory (especially with 4.2.3)
> > > >
> > > > I got in configured and distributed to this user and she is
> > > > still having the same problem. Any other ideas? I double
> > > > checked my code, and it looks like everything is correct. (She
> > > > is not using a
> > > > proxy.)
> > > >
> > >
> > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > >
> > > Oleg
> > >
> > >
> > > >
> > > > Method that makes the request (from multiple threads)
> > > > {
> > > > response = httpClient.execute(request);
> > > > }
> > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > {
> > > > HttpParams params = new BasicHttpParams();
> > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > configureProxy(params,protocol);
> > > >
> > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > }
> > > > private void configureProxy(HttpParams params, String scheme) {
> > > > Proxy proxy = getProxy();
> > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > //SSL or not, we set the proxy up as HTTP
> > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > }
> > > > }
> > > >
> > > > }
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > >
> > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > >
> > > > > I noticed that there is a stale connection check:
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CH
> > > > > EC
> > > > > K,
> > > > > Boolean.TRUE);
> > > > >
> > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > >
> > > > > Mark
> > > > >
> > > >
> > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > To: 'HttpClient User Discussion'
> > > > > Subject: Connection Reset errors
> > > > >
> > > > > I have a user getting a lot of Connection Reset errors. I did
> > > > > not think this had do to with HttpClient, so much as other
> > > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > >
> > > > > This user is fine for a while, and then will get the
> > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > <Connector
> > > > > port="5502"
> > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > server="Unknown"
> > > > > connectionLinger="0"
> > > > > socket.soTimeout="300000"
> > > > > SSLEnabled="true"
> > > > > maxThreads="150"
> > > > > scheme="https"
> > > > > secure="true"
> > > > > clientAuth="false"
> > > > > keystoreFile="-----"
> > > > > keystoreType="PKCS12"
> > > > > keystorePass="-----"
> > > > > sslProtocol="TLS"
> > > > >
> > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA
> > > > > ,T LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3D
> > > > > ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > >
> > > > >
> > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > Message (Connection reset)
> > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > at
> > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > ja
> > > > > va:138)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
This is the longest she has gone without getting the error, so I am calling it fixed. I learned a lot. And, in going to 4.2.3, it is going noticeable faster now. It was hard getting here, but everything is good now. Thanks again!
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 4:55 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
> It looks like all this misery was self inflicted.
Thank you so much for your help. (Setting the linger to 0 was not my idea...)
> I had thought I had tried to change the linger in an earlier test
If only I would have double checked myself on this one.
The user in question has not had time to test it yet; tomorrow I will know for sure. Hopefully what caused the error in my test cases also caused the problem for her.
Mark
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 4:30 PM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Tue, 2013-01-29 at 15:27 -0500, Mark Claassen wrote:
> Wow, that certainly helped my test case! Now to try it for real.
>
> This leaves me with 2 big questions:
> 1) Is there a reason for Tomcat to behave this way?
In the early days of HTTP/1.1 many HTTP agents had issues with persistent connection support. This feature was most likely intended to help broken clients recover by forcing them to open a new connection after a certain number of requests.
> 2) Why is this not WAY more common?
>
> I think the answer to #2 might be (at least partially) that we have our connectionLinger down to 0 to reduce the number of sockets in TIME_WAIT. So, when Tomcat closed the connection, if the client was a bit too slow, the socket would close before the client could finish reading all the data.
>
That explains it. By setting SO_LINGER to zero you are basically instructing Tomcat to terminate connections abnormally by sending a RST (connection reset) command instead of a normal TCP termination procedure.
It looks like all this misery was self inflicted.
Oleg
> I had thought I had tried to change the linger in an earlier test, but maybe I forgot to save the change or something.
>
> In any event, she is trying this and we will see how it goes.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 3:12 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Thanks! I will check it out!
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Tuesday, January 29, 2013 3:09 PM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote:
> > CLUE!
> >
> > I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
> >
> > I about doubled the size of the message I send, and it makes no difference.
> >
> > I put the iteration count in the HttpHeader so I could track it in
> > the Tomcat access log. Tomcat shows that it receives this message
> > (message 100) and that it thinks it succeeded (status == 200)
> >
> > The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
> >
> >
>
>
> Tomcat by default closes persistent connections once it has been used to serve 100 requests.
>
> See 'maxKeepAliveRequests' parameter:
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
>
> Try setting this value to -1 and see if that makes the problem go away.
>
> Oleg
>
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 2:01 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
> >
> > What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
> >
> > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
> > FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
> > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
> > FINE: << " </BODY>[\n]"
> > Thread-00, setSoTimeout(45000) called
> >
> > [snip]
> >
> > Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]:
> > length = 176
> >
> > [snip]
> >
> > [Raw read]: length = 5
> > 0000: 17 03 01 03 0A .....
> > Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
> > 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
> > 0010: C8 08 ..
> > Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception
> > sending alert: java.net.SocketException: Connection reset by peer:
> > socket write error Thread-00, called closeSocket() Thread-00, called
> > close() Thread-00:Jan 29, 2013 1:48:32 PM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 12:39 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Don't know if this will help either...but here is hoping!
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Tuesday, January 29, 2013 9:03 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> > > I ran my test program using 4.01 and it seems to fail even faster.
> > > (Switching a few things around to get it to compile, or course.
> > > Using
> > > ThreadSafeClientConnManager.)
> > >
> > > This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
> > >
> > > Still, the Java.net stuff is completely error free. (I can't say
> > > for sure if she would notice automatic retries, if they occurred.)
> > >
> > > Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
> > >
> >
> > Mark,
> >
> > This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
> > So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
> >
> > So, running your application with SSL debugging turned on should be
> > more
> > informative:
> >
> > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDeb
> > ug
> > .html
> >
> > Oleg
> >
> > > ------------------------------------
> > > Stack trace and debug using 4.0.1
> > >
> > > FINE: Stale connection check
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Attempt 1 to execute request Thread-00:Jan 28, 2013 12:08:45
> > > PM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.client.DefaultRequestDirector execute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection shut down
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> > > releaseConnection
> > > FINE: Released connection is not reusable.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> > > FINE: Releasing connection
> > > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute
> > > notifyWaitingThread
> > > FINE: Notifying no-one, there are no waiting threads
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> > > at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> > > at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> > > at java.lang.Thread.run(Thread.java:662)
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 12:30 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
> > >
> > > The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
> > >
> > > First call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > HttpRequestExecutor.closeConnection:144
> > > HttpRequestExecutor.execute:131
> > > DefaultRequestDirector.tryExecute:717
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Second call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > DefaultRequestDirector.tryExecute:723
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Third call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > HttpPoolEntry.close:89
> > > AbstractConnPool.release:323
> > > PoolingClientConnectionManager.releaseConnection:278
> > > ManagedClientConnectionImpl.abortConnection:463
> > > DefaultRequestDirector.abortConnection:1157
> > > DefaultRequestDirector.execute:621
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:41 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
> > >
> > > Below is the last part of what I got.
> > >
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> > > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > closeExpiredConnections
> > > FINE: Closing expired connections
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > requestConnection
> > > FINE: Connection request: [route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > leaseConnection
> > > FINE: Connection leased: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Stale connection check
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache
> > > process
> > > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestTargetAuthentication
> > > process
> > > FINE: Target auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestProxyAuthentication process
> > > FINE: Proxy auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > > tryExecute
> > > FINE: Attempt 1 to execute request Thread-00:Jan 25, 2013 11:38:01
> > > AM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 0 of 1; total allocated: 0 of 1]
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> > > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> > > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> > > at java.lang.Thread.run(Thread.java:662)
> > > Java Result: 1
> > > BUILD SUCCESSFUL (total time: 26 seconds) -----Original
> > > Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:25 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:07 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Before the upgrade, she was using HttpClient 4.01.
> > > ---
> > > Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
> > >
> > > The website it is connecting to is just a Tomcat server running
> > > the default ROOT webapp. It is delivering the default HTML page
> > > we have set up on that. I would imagine that it is ignoring the
> > > POST and treating it like a GET
> > >
> > > In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 25, 2013 7:46 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > Mark
> > >
> > > You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
> > >
> > > There are probably several things you might want to try out
> > >
> > > (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
> > >
> > > (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
> > >
> > > (3) Otherwise there is no way around getting very close and personal with Wireshark.
> > >
> > > Oleg
> > >
> > >
> > >
> > > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > > > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> > > >
> > > > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Thursday, January 24, 2013 10:29 AM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> > > >
> > > > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> > > >
> > > > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > > > --
> > > >
> > > > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > > > I am trying to force the issue doing a lot of requests all at once.
> > > > I reduced my MAX down to 1 to force contention. Everything
> > > > seems to work pretty well. However, I did get the Connection
> > > > Reset error twice on my machine! The logs were fairly regular
> > > > looking since there is only one connection at a time. However,
> > > > when this happened, it looks like it is sending 2 request at
> > > > once. This shouldn't be happening because there is only one connection.
> > > > (Granted, it is hard to tell if the logging is indicating a
> > > > threading issue, or if the "Connection Reset" error caused the
> > > > logging abnormality.)
> > > >
> > > > I don't need to block around the "httpClient.execute(request);" do I?
> > > >
> > > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > > > --
> > > >
> > > > I do this to establish the connection:
> > > >
> > > > This is run once in the setup
> > > > -----------------------------
> > > > connectionManager = new
> > > > PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > >
> > > >
> > > >
> > > > This is called when I want to make a request
> > > > --------------------------------------------
> > > > response = httpClient.execute(request);
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 5:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Well, the user went all day without any errors while using the Java.net connector.
> > > >
> > > > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> > > >
> > > > Mark
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:47 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Oh, I forgot to add that all my requests are POST requests.
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:02 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> > > >
> > > > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> > > >
> > > > Mark
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 12:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> > > >
> > > > > It may well be that HttpURLConnection silently retries failed
> > > > > requests
> > > > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> > > >
> > > > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> > > >
> > > > public boolean isStale() {
> > > > if (!isOpen()) {
> > > > return true;
> > > > }
> > > > if (isEof()) {
> > > > return true;
> > > > }
> > > > try {
> > > > this.inbuffer.isDataAvailable(1);
> > > > return isEof();
> > > > } catch (SocketTimeoutException ex) {
> > > > return false;
> > > > } catch (IOException ex) {
> > > > return true;
> > > > }
> > > > }
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Friday, January 18, 2013 11:24 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > > > >
> > > >
> > > > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> > > >
> > > >
> > > > > Some other things of note:
> > > > > - She is using JRE 6 Update 13. (There is a long story behind
> > > > > this.)
> > > > > - The application is launched via Webstart
> > > > > - The SocketFactory is created with: fact = new
> > > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory
> > > > > ()
> > > > > ,v
> > > > > er
> > > > > if
> > > > > ier);
> > > > >
> > > >
> > > > I do not thing any of these should matter.
> > > >
> > > > Oleg
> > > >
> > > > > -----Original Message-----
> > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > Sent: Friday, January 18, 2013 6:17 AM
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: Connection Reset errors
> > > > >
> > > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > > > Thanks for the explanation.
> > > > > > > pro-actively evicting expired connections and connections
> > > > > > > that have been idle
> > > > > > Seems like just doing the stale check would more reliable,
> > > > > > and speed does not seem to be a factory (especially with
> > > > > > 4.2.3)
> > > > > >
> > > > > > I got in configured and distributed to this user and she is
> > > > > > still having the same problem. Any other ideas? I double
> > > > > > checked my code, and it looks like everything is correct.
> > > > > > (She is not using a
> > > > > > proxy.)
> > > > > >
> > > > >
> > > > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > >
> > > > > > Method that makes the request (from multiple threads)
> > > > > > {
> > > > > > response = httpClient.execute(request);
> > > > > > }
> > > > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > > > {
> > > > > > HttpParams params = new BasicHttpParams();
> > > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > > > configureProxy(params,protocol);
> > > > > >
> > > > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > > > connectionManager.setMaxTotal(MAX);
> > > > > >
> > > > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > > > }
> > > > > > private void configureProxy(HttpParams params, String scheme) {
> > > > > > Proxy proxy = getProxy();
> > > > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > > > //SSL or not, we set the proxy up as HTTP
> > > > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > > > To: HttpClient User Discussion
> > > > > > Subject: Re: Connection Reset errors
> > > > > >
> > > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > > > >
> > > > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > > > >
> > > > > > > I noticed that there is a stale connection check:
> > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTIO
> > > > > > > N_
> > > > > > > CH
> > > > > > > EC
> > > > > > > K,
> > > > > > > Boolean.TRUE);
> > > > > > >
> > > > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > > > >
> > > > > > > Mark
> > > > > > >
> > > > > >
> > > > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > > > To: 'HttpClient User Discussion'
> > > > > > > Subject: Connection Reset errors
> > > > > > >
> > > > > > > I have a user getting a lot of Connection Reset errors. I
> > > > > > > did not think this had do to with HttpClient, so much as
> > > > > > > other factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > > > >
> > > > > > > This user is fine for a while, and then will get the
> > > > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mark
> > > > > > >
> > > > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > > > <Connector
> > > > > > > port="5502"
> > > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > > > server="Unknown"
> > > > > > > connectionLinger="0"
> > > > > > > socket.soTimeout="300000"
> > > > > > > SSLEnabled="true"
> > > > > > > maxThreads="150"
> > > > > > > scheme="https"
> > > > > > > secure="true"
> > > > > > > clientAuth="false"
> > > > > > > keystoreFile="-----"
> > > > > > > keystoreType="PKCS12"
> > > > > > > keystorePass="-----"
> > > > > > > sslProtocol="TLS"
> > > > > > >
> > > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC
> > > > > > > _S
> > > > > > > HA ,T LS _D HE
> > > > > > > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WIT
> > > > > > > H_ 3D ES _E DE _C
> > > > > > > BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > > > >
> > > > > > >
> > > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > > > Message (Connection reset)
> > > > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > > > at
> > > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > > > ja
> > > > > > > va:138)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ----------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > > >
> > > > > > > ----------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
> It looks like all this misery was self inflicted.
Thank you so much for your help. (Setting the linger to 0 was not my idea...)
> I had thought I had tried to change the linger in an earlier test
If only I would have double checked myself on this one.
The user in question has not had time to test it yet; tomorrow I will know for sure. Hopefully what caused the error in my test cases also caused the problem for her.
Mark
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 4:30 PM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Tue, 2013-01-29 at 15:27 -0500, Mark Claassen wrote:
> Wow, that certainly helped my test case! Now to try it for real.
>
> This leaves me with 2 big questions:
> 1) Is there a reason for Tomcat to behave this way?
In the early days of HTTP/1.1 many HTTP agents had issues with persistent connection support. This feature was most likely intended to help broken clients recover by forcing them to open a new connection after a certain number of requests.
> 2) Why is this not WAY more common?
>
> I think the answer to #2 might be (at least partially) that we have our connectionLinger down to 0 to reduce the number of sockets in TIME_WAIT. So, when Tomcat closed the connection, if the client was a bit too slow, the socket would close before the client could finish reading all the data.
>
That explains it. By setting SO_LINGER to zero you are basically instructing Tomcat to terminate connections abnormally by sending a RST (connection reset) command instead of a normal TCP termination procedure.
It looks like all this misery was self inflicted.
Oleg
> I had thought I had tried to change the linger in an earlier test, but maybe I forgot to save the change or something.
>
> In any event, she is trying this and we will see how it goes.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 3:12 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Thanks! I will check it out!
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Tuesday, January 29, 2013 3:09 PM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote:
> > CLUE!
> >
> > I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
> >
> > I about doubled the size of the message I send, and it makes no difference.
> >
> > I put the iteration count in the HttpHeader so I could track it in
> > the Tomcat access log. Tomcat shows that it receives this message
> > (message 100) and that it thinks it succeeded (status == 200)
> >
> > The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
> >
> >
>
>
> Tomcat by default closes persistent connections once it has been used to serve 100 requests.
>
> See 'maxKeepAliveRequests' parameter:
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
>
> Try setting this value to -1 and see if that makes the problem go away.
>
> Oleg
>
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 2:01 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
> >
> > What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
> >
> > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
> > FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
> > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
> > FINE: << " </BODY>[\n]"
> > Thread-00, setSoTimeout(45000) called
> >
> > [snip]
> >
> > Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]:
> > length = 176
> >
> > [snip]
> >
> > [Raw read]: length = 5
> > 0000: 17 03 01 03 0A .....
> > Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
> > 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
> > 0010: C8 08 ..
> > Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception
> > sending alert: java.net.SocketException: Connection reset by peer:
> > socket write error Thread-00, called closeSocket() Thread-00, called
> > close() Thread-00:Jan 29, 2013 1:48:32 PM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 12:39 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Don't know if this will help either...but here is hoping!
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Tuesday, January 29, 2013 9:03 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> > > I ran my test program using 4.01 and it seems to fail even faster.
> > > (Switching a few things around to get it to compile, or course.
> > > Using
> > > ThreadSafeClientConnManager.)
> > >
> > > This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
> > >
> > > Still, the Java.net stuff is completely error free. (I can't say
> > > for sure if she would notice automatic retries, if they occurred.)
> > >
> > > Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
> > >
> >
> > Mark,
> >
> > This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
> > So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
> >
> > So, running your application with SSL debugging turned on should be
> > more
> > informative:
> >
> > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDeb
> > ug
> > .html
> >
> > Oleg
> >
> > > ------------------------------------
> > > Stack trace and debug using 4.0.1
> > >
> > > FINE: Stale connection check
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Attempt 1 to execute request Thread-00:Jan 28, 2013 12:08:45
> > > PM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.client.DefaultRequestDirector execute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection shut down
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> > > releaseConnection
> > > FINE: Released connection is not reusable.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> > > FINE: Releasing connection
> > > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute
> > > notifyWaitingThread
> > > FINE: Notifying no-one, there are no waiting threads
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> > > at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> > > at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> > > at java.lang.Thread.run(Thread.java:662)
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 12:30 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
> > >
> > > The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
> > >
> > > First call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > HttpRequestExecutor.closeConnection:144
> > > HttpRequestExecutor.execute:131
> > > DefaultRequestDirector.tryExecute:717
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Second call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > DefaultRequestDirector.tryExecute:723
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Third call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > HttpPoolEntry.close:89
> > > AbstractConnPool.release:323
> > > PoolingClientConnectionManager.releaseConnection:278
> > > ManagedClientConnectionImpl.abortConnection:463
> > > DefaultRequestDirector.abortConnection:1157
> > > DefaultRequestDirector.execute:621
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:41 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
> > >
> > > Below is the last part of what I got.
> > >
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> > > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > closeExpiredConnections
> > > FINE: Closing expired connections
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > requestConnection
> > > FINE: Connection request: [route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > leaseConnection
> > > FINE: Connection leased: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Stale connection check
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache
> > > process
> > > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestTargetAuthentication
> > > process
> > > FINE: Target auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestProxyAuthentication process
> > > FINE: Proxy auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > > tryExecute
> > > FINE: Attempt 1 to execute request Thread-00:Jan 25, 2013 11:38:01
> > > AM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 0 of 1; total allocated: 0 of 1]
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> > > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> > > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> > > at java.lang.Thread.run(Thread.java:662)
> > > Java Result: 1
> > > BUILD SUCCESSFUL (total time: 26 seconds) -----Original
> > > Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:25 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:07 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Before the upgrade, she was using HttpClient 4.01.
> > > ---
> > > Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
> > >
> > > The website it is connecting to is just a Tomcat server running
> > > the default ROOT webapp. It is delivering the default HTML page
> > > we have set up on that. I would imagine that it is ignoring the
> > > POST and treating it like a GET
> > >
> > > In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 25, 2013 7:46 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > Mark
> > >
> > > You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
> > >
> > > There are probably several things you might want to try out
> > >
> > > (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
> > >
> > > (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
> > >
> > > (3) Otherwise there is no way around getting very close and personal with Wireshark.
> > >
> > > Oleg
> > >
> > >
> > >
> > > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > > > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> > > >
> > > > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Thursday, January 24, 2013 10:29 AM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> > > >
> > > > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> > > >
> > > > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > > > --
> > > >
> > > > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > > > I am trying to force the issue doing a lot of requests all at once.
> > > > I reduced my MAX down to 1 to force contention. Everything
> > > > seems to work pretty well. However, I did get the Connection
> > > > Reset error twice on my machine! The logs were fairly regular
> > > > looking since there is only one connection at a time. However,
> > > > when this happened, it looks like it is sending 2 request at
> > > > once. This shouldn't be happening because there is only one connection.
> > > > (Granted, it is hard to tell if the logging is indicating a
> > > > threading issue, or if the "Connection Reset" error caused the
> > > > logging abnormality.)
> > > >
> > > > I don't need to block around the "httpClient.execute(request);" do I?
> > > >
> > > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > > > --
> > > >
> > > > I do this to establish the connection:
> > > >
> > > > This is run once in the setup
> > > > -----------------------------
> > > > connectionManager = new
> > > > PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > >
> > > >
> > > >
> > > > This is called when I want to make a request
> > > > --------------------------------------------
> > > > response = httpClient.execute(request);
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 5:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Well, the user went all day without any errors while using the Java.net connector.
> > > >
> > > > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> > > >
> > > > Mark
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:47 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Oh, I forgot to add that all my requests are POST requests.
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:02 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> > > >
> > > > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> > > >
> > > > Mark
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 12:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> > > >
> > > > > It may well be that HttpURLConnection silently retries failed
> > > > > requests
> > > > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> > > >
> > > > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> > > >
> > > > public boolean isStale() {
> > > > if (!isOpen()) {
> > > > return true;
> > > > }
> > > > if (isEof()) {
> > > > return true;
> > > > }
> > > > try {
> > > > this.inbuffer.isDataAvailable(1);
> > > > return isEof();
> > > > } catch (SocketTimeoutException ex) {
> > > > return false;
> > > > } catch (IOException ex) {
> > > > return true;
> > > > }
> > > > }
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Friday, January 18, 2013 11:24 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > > > >
> > > >
> > > > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> > > >
> > > >
> > > > > Some other things of note:
> > > > > - She is using JRE 6 Update 13. (There is a long story behind
> > > > > this.)
> > > > > - The application is launched via Webstart
> > > > > - The SocketFactory is created with: fact = new
> > > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory
> > > > > ()
> > > > > ,v
> > > > > er
> > > > > if
> > > > > ier);
> > > > >
> > > >
> > > > I do not thing any of these should matter.
> > > >
> > > > Oleg
> > > >
> > > > > -----Original Message-----
> > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > Sent: Friday, January 18, 2013 6:17 AM
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: Connection Reset errors
> > > > >
> > > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > > > Thanks for the explanation.
> > > > > > > pro-actively evicting expired connections and connections
> > > > > > > that have been idle
> > > > > > Seems like just doing the stale check would more reliable,
> > > > > > and speed does not seem to be a factory (especially with
> > > > > > 4.2.3)
> > > > > >
> > > > > > I got in configured and distributed to this user and she is
> > > > > > still having the same problem. Any other ideas? I double
> > > > > > checked my code, and it looks like everything is correct.
> > > > > > (She is not using a
> > > > > > proxy.)
> > > > > >
> > > > >
> > > > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > >
> > > > > > Method that makes the request (from multiple threads)
> > > > > > {
> > > > > > response = httpClient.execute(request);
> > > > > > }
> > > > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > > > {
> > > > > > HttpParams params = new BasicHttpParams();
> > > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > > > configureProxy(params,protocol);
> > > > > >
> > > > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > > > connectionManager.setMaxTotal(MAX);
> > > > > >
> > > > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > > > }
> > > > > > private void configureProxy(HttpParams params, String scheme) {
> > > > > > Proxy proxy = getProxy();
> > > > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > > > //SSL or not, we set the proxy up as HTTP
> > > > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > > > To: HttpClient User Discussion
> > > > > > Subject: Re: Connection Reset errors
> > > > > >
> > > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > > > >
> > > > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > > > >
> > > > > > > I noticed that there is a stale connection check:
> > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTIO
> > > > > > > N_
> > > > > > > CH
> > > > > > > EC
> > > > > > > K,
> > > > > > > Boolean.TRUE);
> > > > > > >
> > > > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > > > >
> > > > > > > Mark
> > > > > > >
> > > > > >
> > > > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > > > To: 'HttpClient User Discussion'
> > > > > > > Subject: Connection Reset errors
> > > > > > >
> > > > > > > I have a user getting a lot of Connection Reset errors. I
> > > > > > > did not think this had do to with HttpClient, so much as
> > > > > > > other factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > > > >
> > > > > > > This user is fine for a while, and then will get the
> > > > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mark
> > > > > > >
> > > > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > > > <Connector
> > > > > > > port="5502"
> > > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > > > server="Unknown"
> > > > > > > connectionLinger="0"
> > > > > > > socket.soTimeout="300000"
> > > > > > > SSLEnabled="true"
> > > > > > > maxThreads="150"
> > > > > > > scheme="https"
> > > > > > > secure="true"
> > > > > > > clientAuth="false"
> > > > > > > keystoreFile="-----"
> > > > > > > keystoreType="PKCS12"
> > > > > > > keystorePass="-----"
> > > > > > > sslProtocol="TLS"
> > > > > > >
> > > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC
> > > > > > > _S
> > > > > > > HA ,T LS _D HE
> > > > > > > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WIT
> > > > > > > H_ 3D ES _E DE _C
> > > > > > > BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > > > >
> > > > > > >
> > > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > > > Message (Connection reset)
> > > > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > > > at
> > > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > > > ja
> > > > > > > va:138)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ----------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > > >
> > > > > > > ----------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2013-01-29 at 15:27 -0500, Mark Claassen wrote:
> Wow, that certainly helped my test case! Now to try it for real.
>
> This leaves me with 2 big questions:
> 1) Is there a reason for Tomcat to behave this way?
In the early days of HTTP/1.1 many HTTP agents had issues with
persistent connection support. This feature was most likely intended to
help broken clients recover by forcing them to open a new connection
after a certain number of requests.
> 2) Why is this not WAY more common?
>
> I think the answer to #2 might be (at least partially) that we have our connectionLinger down to 0 to reduce the number of sockets in TIME_WAIT. So, when Tomcat closed the connection, if the client was a bit too slow, the socket would close before the client could finish reading all the data.
>
That explains it. By setting SO_LINGER to zero you are basically
instructing Tomcat to terminate connections abnormally by sending a RST
(connection reset) command instead of a normal TCP termination
procedure.
It looks like all this misery was self inflicted.
Oleg
> I had thought I had tried to change the linger in an earlier test, but maybe I forgot to save the change or something.
>
> In any event, she is trying this and we will see how it goes.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 3:12 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Thanks! I will check it out!
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Tuesday, January 29, 2013 3:09 PM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote:
> > CLUE!
> >
> > I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
> >
> > I about doubled the size of the message I send, and it makes no difference.
> >
> > I put the iteration count in the HttpHeader so I could track it in the
> > Tomcat access log. Tomcat shows that it receives this message
> > (message 100) and that it thinks it succeeded (status == 200)
> >
> > The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
> >
> >
>
>
> Tomcat by default closes persistent connections once it has been used to serve 100 requests.
>
> See 'maxKeepAliveRequests' parameter:
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
>
> Try setting this value to -1 and see if that makes the problem go away.
>
> Oleg
>
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 2:01 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
> >
> > What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
> >
> > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
> > FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
> > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
> > FINE: << " </BODY>[\n]"
> > Thread-00, setSoTimeout(45000) called
> >
> > [snip]
> >
> > Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]:
> > length = 176
> >
> > [snip]
> >
> > [Raw read]: length = 5
> > 0000: 17 03 01 03 0A .....
> > Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
> > 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
> > 0010: C8 08 ..
> > Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception
> > sending alert: java.net.SocketException: Connection reset by peer:
> > socket write error Thread-00, called closeSocket() Thread-00, called
> > close() Thread-00:Jan 29, 2013 1:48:32 PM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Tuesday, January 29, 2013 12:39 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Don't know if this will help either...but here is hoping!
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Tuesday, January 29, 2013 9:03 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> > > I ran my test program using 4.01 and it seems to fail even faster.
> > > (Switching a few things around to get it to compile, or course.
> > > Using
> > > ThreadSafeClientConnManager.)
> > >
> > > This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
> > >
> > > Still, the Java.net stuff is completely error free. (I can't say
> > > for sure if she would notice automatic retries, if they occurred.)
> > >
> > > Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
> > >
> >
> > Mark,
> >
> > This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
> > So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
> >
> > So, running your application with SSL debugging turned on should be
> > more
> > informative:
> >
> > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug
> > .html
> >
> > Oleg
> >
> > > ------------------------------------
> > > Stack trace and debug using 4.0.1
> > >
> > > FINE: Stale connection check
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Attempt 1 to execute request
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[EOL]"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> > > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.client.DefaultRequestDirector execute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection closed
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection shut down
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> > > releaseConnection
> > > FINE: Released connection is not reusable.
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> > > FINE: Releasing connection
> > > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> > > Thread-00:Jan 28, 2013 12:08:45 PM
> > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> > > FINE: Notifying no-one, there are no waiting threads
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> > > at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> > > at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> > > at java.lang.Thread.run(Thread.java:662)
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 12:30 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
> > >
> > > The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
> > >
> > > First call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > HttpRequestExecutor.closeConnection:144
> > > HttpRequestExecutor.execute:131
> > > DefaultRequestDirector.tryExecute:717
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Second call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > ManagedClientConnectionImpl.close:127
> > > DefaultRequestDirector.tryExecute:723
> > > DefaultRequestDirector.execute:522
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > > Third call to close()
> > > ----------------------
> > > DefaultClientConnection.close:168
> > > HttpPoolEntry.close:89
> > > AbstractConnPool.release:323
> > > PoolingClientConnectionManager.releaseConnection:278
> > > ManagedClientConnectionImpl.abortConnection:463
> > > DefaultRequestDirector.abortConnection:1157
> > > DefaultRequestDirector.execute:621
> > > AbstractHttpClient.execute:906
> > > AbstractHttpClient.execute:805
> > > AbstractHttpClient.execute:784
> > > TestNetwork$Worker.connect:187
> > > TestNetwork$Worker.run:153
> > > Thread.run:662
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:41 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
> > >
> > > Below is the last part of what I got.
> > >
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> > > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > > closeExpiredConnections
> > > FINE: Closing expired connections
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > requestConnection
> > > FINE: Connection request: [route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > > leaseConnection
> > > FINE: Connection leased: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > > execute
> > > FINE: Stale connection check
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.client.protocol.RequestAddCookies process
> > > FINE: CookieSpec selected: best-match Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process
> > > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestTargetAuthentication process
> > > FINE: Target auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > > 11:38:01 AM
> > > org.apache.http.client.protocol.RequestProxyAuthentication process
> > > FINE: Proxy auth state: UNCHALLENGED Thread-00:Jan 25, 2013 11:38:01
> > > AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > > FINE: Attempt 1 to execute request
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > > FINE: >> "POST / HTTP/1.1[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Type: binary[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Content-Length: 34[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "Connection: Keep-Alive[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "[\r][\n]"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> POST / HTTP/1.1
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> Content-Type: binary
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> Content-Length: 34
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> > > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> > > sendRequestHeader
> > > FINE: >> Connection: Keep-Alive
> > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > > wire
> > > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > > FINE: Closing the connection.
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.DefaultClientConnection close
> > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > > Thread-00:Jan 25, 2013 11:38:01 AM
> > > org.apache.http.impl.conn.PoolingClientConnectionManager
> > > releaseConnection
> > > FINE: Connection released: [id: 0][route:
> > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > > route
> > > allocated: 0 of 1; total allocated: 0 of 1]
> > > java.net.SocketException: Connection reset
> > > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> > > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> > > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> > > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> > > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> > > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> > > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> > > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> > > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> > > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> > > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> > > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> > > at java.lang.Thread.run(Thread.java:662)
> > > Java Result: 1
> > > BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:25 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 25, 2013 11:07 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Before the upgrade, she was using HttpClient 4.01.
> > > ---
> > > Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
> > >
> > > The website it is connecting to is just a Tomcat server running the
> > > default ROOT webapp. It is delivering the default HTML page we have
> > > set up on that. I would imagine that it is ignoring the POST and
> > > treating it like a GET
> > >
> > > In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 25, 2013 7:46 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > Mark
> > >
> > > You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
> > >
> > > There are probably several things you might want to try out
> > >
> > > (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
> > >
> > > (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
> > >
> > > (3) Otherwise there is no way around getting very close and personal with Wireshark.
> > >
> > > Oleg
> > >
> > >
> > >
> > > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > > > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> > > >
> > > > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Thursday, January 24, 2013 10:29 AM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> > > >
> > > > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> > > >
> > > > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > > > --
> > > >
> > > > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > > > I am trying to force the issue doing a lot of requests all at once.
> > > > I reduced my MAX down to 1 to force contention. Everything seems
> > > > to work pretty well. However, I did get the Connection Reset
> > > > error twice on my machine! The logs were fairly regular looking
> > > > since there is only one connection at a time. However, when this
> > > > happened, it looks like it is sending 2 request at once. This
> > > > shouldn't be happening because there is only one connection.
> > > > (Granted, it is hard to tell if the logging is indicating a
> > > > threading issue, or if the "Connection Reset" error caused the
> > > > logging abnormality.)
> > > >
> > > > I don't need to block around the "httpClient.execute(request);" do I?
> > > >
> > > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > > > --
> > > >
> > > > I do this to establish the connection:
> > > >
> > > > This is run once in the setup
> > > > -----------------------------
> > > > connectionManager = new
> > > > PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > >
> > > >
> > > >
> > > > This is called when I want to make a request
> > > > --------------------------------------------
> > > > response = httpClient.execute(request);
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 5:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Well, the user went all day without any errors while using the Java.net connector.
> > > >
> > > > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> > > >
> > > > Mark
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:47 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Oh, I forgot to add that all my requests are POST requests.
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 1:02 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> > > >
> > > > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> > > >
> > > > Mark
> > > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Friday, January 18, 2013 12:06 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: RE: Connection Reset errors
> > > >
> > > > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> > > >
> > > > > It may well be that HttpURLConnection silently retries failed
> > > > > requests
> > > > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> > > >
> > > > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> > > >
> > > > public boolean isStale() {
> > > > if (!isOpen()) {
> > > > return true;
> > > > }
> > > > if (isEof()) {
> > > > return true;
> > > > }
> > > > try {
> > > > this.inbuffer.isDataAvailable(1);
> > > > return isEof();
> > > > } catch (SocketTimeoutException ex) {
> > > > return false;
> > > > } catch (IOException ex) {
> > > > return true;
> > > > }
> > > > }
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Friday, January 18, 2013 11:24 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > > > >
> > > >
> > > > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> > > >
> > > >
> > > > > Some other things of note:
> > > > > - She is using JRE 6 Update 13. (There is a long story behind
> > > > > this.)
> > > > > - The application is launched via Webstart
> > > > > - The SocketFactory is created with: fact = new
> > > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory()
> > > > > ,v
> > > > > er
> > > > > if
> > > > > ier);
> > > > >
> > > >
> > > > I do not thing any of these should matter.
> > > >
> > > > Oleg
> > > >
> > > > > -----Original Message-----
> > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > Sent: Friday, January 18, 2013 6:17 AM
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: Connection Reset errors
> > > > >
> > > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > > > Thanks for the explanation.
> > > > > > > pro-actively evicting expired connections and connections
> > > > > > > that have been idle
> > > > > > Seems like just doing the stale check would more reliable, and
> > > > > > speed does not seem to be a factory (especially with 4.2.3)
> > > > > >
> > > > > > I got in configured and distributed to this user and she is
> > > > > > still having the same problem. Any other ideas? I double
> > > > > > checked my code, and it looks like everything is correct.
> > > > > > (She is not using a
> > > > > > proxy.)
> > > > > >
> > > > >
> > > > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > >
> > > > > > Method that makes the request (from multiple threads)
> > > > > > {
> > > > > > response = httpClient.execute(request);
> > > > > > }
> > > > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > > > {
> > > > > > HttpParams params = new BasicHttpParams();
> > > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > > > configureProxy(params,protocol);
> > > > > >
> > > > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > > > connectionManager.setMaxTotal(MAX);
> > > > > >
> > > > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > > > }
> > > > > > private void configureProxy(HttpParams params, String scheme) {
> > > > > > Proxy proxy = getProxy();
> > > > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > > > //SSL or not, we set the proxy up as HTTP
> > > > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > }
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > > > To: HttpClient User Discussion
> > > > > > Subject: Re: Connection Reset errors
> > > > > >
> > > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > > > >
> > > > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > > > >
> > > > > > > I noticed that there is a stale connection check:
> > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_
> > > > > > > CH
> > > > > > > EC
> > > > > > > K,
> > > > > > > Boolean.TRUE);
> > > > > > >
> > > > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > > > >
> > > > > > > Mark
> > > > > > >
> > > > > >
> > > > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > > > >
> > > > > > Oleg
> > > > > >
> > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > > > To: 'HttpClient User Discussion'
> > > > > > > Subject: Connection Reset errors
> > > > > > >
> > > > > > > I have a user getting a lot of Connection Reset errors. I
> > > > > > > did not think this had do to with HttpClient, so much as
> > > > > > > other factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > > > >
> > > > > > > This user is fine for a while, and then will get the
> > > > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mark
> > > > > > >
> > > > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > > > <Connector
> > > > > > > port="5502"
> > > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > > > server="Unknown"
> > > > > > > connectionLinger="0"
> > > > > > > socket.soTimeout="300000"
> > > > > > > SSLEnabled="true"
> > > > > > > maxThreads="150"
> > > > > > > scheme="https"
> > > > > > > secure="true"
> > > > > > > clientAuth="false"
> > > > > > > keystoreFile="-----"
> > > > > > > keystoreType="PKCS12"
> > > > > > > keystorePass="-----"
> > > > > > > sslProtocol="TLS"
> > > > > > >
> > > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_S
> > > > > > > HA ,T LS _D HE
> > > > > > > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_
> > > > > > > 3D ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > > > >
> > > > > > >
> > > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > > > Message (Connection reset)
> > > > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > > > at
> > > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > > > ja
> > > > > > > va:138)
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > > >
> > > > > > > ------------------------------------------------------------
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > --
> > > > > > > - To unsubscribe, e-mail:
> > > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > > For additional commands, e-mail:
> > > > > > > httpclient-users-help@hc.apache.org
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > ----------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Wow, that certainly helped my test case! Now to try it for real.
This leaves me with 2 big questions:
1) Is there a reason for Tomcat to behave this way?
2) Why is this not WAY more common?
I think the answer to #2 might be (at least partially) that we have our connectionLinger down to 0 to reduce the number of sockets in TIME_WAIT. So, when Tomcat closed the connection, if the client was a bit too slow, the socket would close before the client could finish reading all the data.
I had thought I had tried to change the linger in an earlier test, but maybe I forgot to save the change or something.
In any event, she is trying this and we will see how it goes.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 3:12 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Thanks! I will check it out!
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 3:09 PM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote:
> CLUE!
>
> I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
>
> I about doubled the size of the message I send, and it makes no difference.
>
> I put the iteration count in the HttpHeader so I could track it in the
> Tomcat access log. Tomcat shows that it receives this message
> (message 100) and that it thinks it succeeded (status == 200)
>
> The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
>
>
Tomcat by default closes persistent connections once it has been used to serve 100 requests.
See 'maxKeepAliveRequests' parameter:
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Try setting this value to -1 and see if that makes the problem go away.
Oleg
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 2:01 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
>
> What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
>
> Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
> FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
> Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
> FINE: << " </BODY>[\n]"
> Thread-00, setSoTimeout(45000) called
>
> [snip]
>
> Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]:
> length = 176
>
> [snip]
>
> [Raw read]: length = 5
> 0000: 17 03 01 03 0A .....
> Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
> 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
> 0010: C8 08 ..
> Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception
> sending alert: java.net.SocketException: Connection reset by peer:
> socket write error Thread-00, called closeSocket() Thread-00, called
> close() Thread-00:Jan 29, 2013 1:48:32 PM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 12:39 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Don't know if this will help either...but here is hoping!
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Tuesday, January 29, 2013 9:03 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> > I ran my test program using 4.01 and it seems to fail even faster.
> > (Switching a few things around to get it to compile, or course.
> > Using
> > ThreadSafeClientConnManager.)
> >
> > This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
> >
> > Still, the Java.net stuff is completely error free. (I can't say
> > for sure if she would notice automatic retries, if they occurred.)
> >
> > Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
> >
>
> Mark,
>
> This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
> So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
>
> So, running your application with SSL debugging turned on should be
> more
> informative:
>
> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug
> .html
>
> Oleg
>
> > ------------------------------------
> > Stack trace and debug using 4.0.1
> >
> > FINE: Stale connection check
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.client.protocol.RequestAddCookies process
> > FINE: CookieSpec selected: best-match Thread-00:Jan 28, 2013
> > 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector
> > execute
> > FINE: Attempt 1 to execute request
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013
> > 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "POST / HTTP/1.1[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Type: binary[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Length: 34[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Connection: Keep-Alive[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> POST / HTTP/1.1
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Type: binary
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Length: 34
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> > sendRequestHeader
> > FINE: >> Connection: Keep-Alive
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection closed
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.client.DefaultRequestDirector execute
> > FINE: Closing the connection.
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection closed
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > FINE: Connection shut down
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> > releaseConnection
> > FINE: Released connection is not reusable.
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> > FINE: Releasing connection
> > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> > FINE: Notifying no-one, there are no waiting threads
> > java.net.SocketException: Connection reset
> > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> > at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> > at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 12:30 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
> >
> > The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
> >
> > First call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > ManagedClientConnectionImpl.close:127
> > HttpRequestExecutor.closeConnection:144
> > HttpRequestExecutor.execute:131
> > DefaultRequestDirector.tryExecute:717
> > DefaultRequestDirector.execute:522
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> > Second call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > ManagedClientConnectionImpl.close:127
> > DefaultRequestDirector.tryExecute:723
> > DefaultRequestDirector.execute:522
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> > Third call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > HttpPoolEntry.close:89
> > AbstractConnPool.release:323
> > PoolingClientConnectionManager.releaseConnection:278
> > ManagedClientConnectionImpl.abortConnection:463
> > DefaultRequestDirector.abortConnection:1157
> > DefaultRequestDirector.execute:621
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:41 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
> >
> > Below is the last part of what I got.
> >
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection released: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > closeExpiredConnections
> > FINE: Closing expired connections
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > requestConnection
> > FINE: Connection request: [route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > leaseConnection
> > FINE: Connection leased: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > execute
> > FINE: Stale connection check
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.client.protocol.RequestAddCookies process
> > FINE: CookieSpec selected: best-match Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process
> > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> > 11:38:01 AM
> > org.apache.http.client.protocol.RequestTargetAuthentication process
> > FINE: Target auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > 11:38:01 AM
> > org.apache.http.client.protocol.RequestProxyAuthentication process
> > FINE: Proxy auth state: UNCHALLENGED Thread-00:Jan 25, 2013 11:38:01
> > AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > FINE: Attempt 1 to execute request
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "POST / HTTP/1.1[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Type: binary[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Length: 34[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Connection: Keep-Alive[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> POST / HTTP/1.1
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Type: binary
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Length: 34
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> > sendRequestHeader
> > FINE: >> Connection: Keep-Alive
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > FINE: Closing the connection.
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection released: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > route
> > allocated: 0 of 1; total allocated: 0 of 1]
> > java.net.SocketException: Connection reset
> > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> > at java.lang.Thread.run(Thread.java:662)
> > Java Result: 1
> > BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:25 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:07 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Before the upgrade, she was using HttpClient 4.01.
> > ---
> > Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
> >
> > The website it is connecting to is just a Tomcat server running the
> > default ROOT webapp. It is delivering the default HTML page we have
> > set up on that. I would imagine that it is ignoring the POST and
> > treating it like a GET
> >
> > In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 25, 2013 7:46 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > Mark
> >
> > You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
> >
> > There are probably several things you might want to try out
> >
> > (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
> >
> > (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
> >
> > (3) Otherwise there is no way around getting very close and personal with Wireshark.
> >
> > Oleg
> >
> >
> >
> > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> > >
> > > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Thursday, January 24, 2013 10:29 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> > >
> > > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> > >
> > > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > > --
> > >
> > > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > > I am trying to force the issue doing a lot of requests all at once.
> > > I reduced my MAX down to 1 to force contention. Everything seems
> > > to work pretty well. However, I did get the Connection Reset
> > > error twice on my machine! The logs were fairly regular looking
> > > since there is only one connection at a time. However, when this
> > > happened, it looks like it is sending 2 request at once. This
> > > shouldn't be happening because there is only one connection.
> > > (Granted, it is hard to tell if the logging is indicating a
> > > threading issue, or if the "Connection Reset" error caused the
> > > logging abnormality.)
> > >
> > > I don't need to block around the "httpClient.execute(request);" do I?
> > >
> > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > > --
> > >
> > > I do this to establish the connection:
> > >
> > > This is run once in the setup
> > > -----------------------------
> > > connectionManager = new
> > > PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > >
> > >
> > >
> > > This is called when I want to make a request
> > > --------------------------------------------
> > > response = httpClient.execute(request);
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 5:06 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Well, the user went all day without any errors while using the Java.net connector.
> > >
> > > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> > >
> > > Mark
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 1:47 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Oh, I forgot to add that all my requests are POST requests.
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 1:02 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> > >
> > > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> > >
> > > Mark
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 12:06 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> > >
> > > > It may well be that HttpURLConnection silently retries failed
> > > > requests
> > > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> > >
> > > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> > >
> > > public boolean isStale() {
> > > if (!isOpen()) {
> > > return true;
> > > }
> > > if (isEof()) {
> > > return true;
> > > }
> > > try {
> > > this.inbuffer.isDataAvailable(1);
> > > return isEof();
> > > } catch (SocketTimeoutException ex) {
> > > return false;
> > > } catch (IOException ex) {
> > > return true;
> > > }
> > > }
> > >
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 11:24 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > > >
> > >
> > > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> > >
> > >
> > > > Some other things of note:
> > > > - She is using JRE 6 Update 13. (There is a long story behind
> > > > this.)
> > > > - The application is launched via Webstart
> > > > - The SocketFactory is created with: fact = new
> > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory()
> > > > ,v
> > > > er
> > > > if
> > > > ier);
> > > >
> > >
> > > I do not thing any of these should matter.
> > >
> > > Oleg
> > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Friday, January 18, 2013 6:17 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > > Thanks for the explanation.
> > > > > > pro-actively evicting expired connections and connections
> > > > > > that have been idle
> > > > > Seems like just doing the stale check would more reliable, and
> > > > > speed does not seem to be a factory (especially with 4.2.3)
> > > > >
> > > > > I got in configured and distributed to this user and she is
> > > > > still having the same problem. Any other ideas? I double
> > > > > checked my code, and it looks like everything is correct.
> > > > > (She is not using a
> > > > > proxy.)
> > > > >
> > > >
> > > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > > >
> > > > Oleg
> > > >
> > > >
> > > > >
> > > > > Method that makes the request (from multiple threads)
> > > > > {
> > > > > response = httpClient.execute(request);
> > > > > }
> > > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > > {
> > > > > HttpParams params = new BasicHttpParams();
> > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > > configureProxy(params,protocol);
> > > > >
> > > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > > connectionManager.setMaxTotal(MAX);
> > > > >
> > > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > > }
> > > > > private void configureProxy(HttpParams params, String scheme) {
> > > > > Proxy proxy = getProxy();
> > > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > > //SSL or not, we set the proxy up as HTTP
> > > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > > }
> > > > > }
> > > > >
> > > > > }
> > > > >
> > > > > -----Original Message-----
> > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: Connection Reset errors
> > > > >
> > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > > >
> > > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > > >
> > > > > > I noticed that there is a stale connection check:
> > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_
> > > > > > CH
> > > > > > EC
> > > > > > K,
> > > > > > Boolean.TRUE);
> > > > > >
> > > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > > >
> > > > > > Mark
> > > > > >
> > > > >
> > > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > > To: 'HttpClient User Discussion'
> > > > > > Subject: Connection Reset errors
> > > > > >
> > > > > > I have a user getting a lot of Connection Reset errors. I
> > > > > > did not think this had do to with HttpClient, so much as
> > > > > > other factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > > >
> > > > > > This user is fine for a while, and then will get the
> > > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > > <Connector
> > > > > > port="5502"
> > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > > server="Unknown"
> > > > > > connectionLinger="0"
> > > > > > socket.soTimeout="300000"
> > > > > > SSLEnabled="true"
> > > > > > maxThreads="150"
> > > > > > scheme="https"
> > > > > > secure="true"
> > > > > > clientAuth="false"
> > > > > > keystoreFile="-----"
> > > > > > keystoreType="PKCS12"
> > > > > > keystorePass="-----"
> > > > > > sslProtocol="TLS"
> > > > > >
> > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_S
> > > > > > HA ,T LS _D HE
> > > > > > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_
> > > > > > 3D ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > > >
> > > > > >
> > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > > Message (Connection reset)
> > > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > > at
> > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > > ja
> > > > > > va:138)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Thanks! I will check it out!
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 3:09 PM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote:
> CLUE!
>
> I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
>
> I about doubled the size of the message I send, and it makes no difference.
>
> I put the iteration count in the HttpHeader so I could track it in the
> Tomcat access log. Tomcat shows that it receives this message
> (message 100) and that it thinks it succeeded (status == 200)
>
> The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
>
>
Tomcat by default closes persistent connections once it has been used to serve 100 requests.
See 'maxKeepAliveRequests' parameter:
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Try setting this value to -1 and see if that makes the problem go away.
Oleg
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 2:01 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
>
> What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
>
> Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
> FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
> Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
> FINE: << " </BODY>[\n]"
> Thread-00, setSoTimeout(45000) called
>
> [snip]
>
> Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]:
> length = 176
>
> [snip]
>
> [Raw read]: length = 5
> 0000: 17 03 01 03 0A .....
> Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
> 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
> 0010: C8 08 ..
> Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception
> sending alert: java.net.SocketException: Connection reset by peer:
> socket write error Thread-00, called closeSocket() Thread-00, called
> close() Thread-00:Jan 29, 2013 1:48:32 PM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 12:39 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Don't know if this will help either...but here is hoping!
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Tuesday, January 29, 2013 9:03 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> > I ran my test program using 4.01 and it seems to fail even faster.
> > (Switching a few things around to get it to compile, or course.
> > Using
> > ThreadSafeClientConnManager.)
> >
> > This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
> >
> > Still, the Java.net stuff is completely error free. (I can't say
> > for sure if she would notice automatic retries, if they occurred.)
> >
> > Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
> >
>
> Mark,
>
> This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
> So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
>
> So, running your application with SSL debugging turned on should be
> more
> informative:
>
> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug
> .html
>
> Oleg
>
> > ------------------------------------
> > Stack trace and debug using 4.0.1
> >
> > FINE: Stale connection check
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.client.protocol.RequestAddCookies process
> > FINE: CookieSpec selected: best-match Thread-00:Jan 28, 2013
> > 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector
> > execute
> > FINE: Attempt 1 to execute request
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013
> > 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "POST / HTTP/1.1[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Type: binary[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Length: 34[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Connection: Keep-Alive[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> POST / HTTP/1.1
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Type: binary
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Length: 34
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> > sendRequestHeader
> > FINE: >> Connection: Keep-Alive
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection closed
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.client.DefaultRequestDirector execute
> > FINE: Closing the connection.
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection closed
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > FINE: Connection shut down
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> > releaseConnection
> > FINE: Released connection is not reusable.
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> > FINE: Releasing connection
> > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> > FINE: Notifying no-one, there are no waiting threads
> > java.net.SocketException: Connection reset
> > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> > at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> > at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 12:30 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
> >
> > The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
> >
> > First call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > ManagedClientConnectionImpl.close:127
> > HttpRequestExecutor.closeConnection:144
> > HttpRequestExecutor.execute:131
> > DefaultRequestDirector.tryExecute:717
> > DefaultRequestDirector.execute:522
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> > Second call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > ManagedClientConnectionImpl.close:127
> > DefaultRequestDirector.tryExecute:723
> > DefaultRequestDirector.execute:522
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> > Third call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > HttpPoolEntry.close:89
> > AbstractConnPool.release:323
> > PoolingClientConnectionManager.releaseConnection:278
> > ManagedClientConnectionImpl.abortConnection:463
> > DefaultRequestDirector.abortConnection:1157
> > DefaultRequestDirector.execute:621
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:41 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
> >
> > Below is the last part of what I got.
> >
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection released: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > closeExpiredConnections
> > FINE: Closing expired connections
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > requestConnection
> > FINE: Connection request: [route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1;
> > route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > leaseConnection
> > FINE: Connection leased: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector
> > execute
> > FINE: Stale connection check
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.client.protocol.RequestAddCookies process
> > FINE: CookieSpec selected: best-match Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process
> > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> > 11:38:01 AM
> > org.apache.http.client.protocol.RequestTargetAuthentication process
> > FINE: Target auth state: UNCHALLENGED Thread-00:Jan 25, 2013
> > 11:38:01 AM
> > org.apache.http.client.protocol.RequestProxyAuthentication process
> > FINE: Proxy auth state: UNCHALLENGED Thread-00:Jan 25, 2013 11:38:01
> > AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > FINE: Attempt 1 to execute request
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "POST / HTTP/1.1[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Type: binary[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Content-Length: 34[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "Connection: Keep-Alive[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> POST / HTTP/1.1
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Type: binary
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Length: 34
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> > sendRequestHeader
> > FINE: >> Connection: Keep-Alive
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire
> > wire
> > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > FINE: Closing the connection.
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection released: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0;
> > route
> > allocated: 0 of 1; total allocated: 0 of 1]
> > java.net.SocketException: Connection reset
> > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> > at java.lang.Thread.run(Thread.java:662)
> > Java Result: 1
> > BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:25 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:07 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Before the upgrade, she was using HttpClient 4.01.
> > ---
> > Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
> >
> > The website it is connecting to is just a Tomcat server running the
> > default ROOT webapp. It is delivering the default HTML page we have
> > set up on that. I would imagine that it is ignoring the POST and
> > treating it like a GET
> >
> > In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 25, 2013 7:46 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > Mark
> >
> > You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
> >
> > There are probably several things you might want to try out
> >
> > (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
> >
> > (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
> >
> > (3) Otherwise there is no way around getting very close and personal with Wireshark.
> >
> > Oleg
> >
> >
> >
> > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> > >
> > > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Thursday, January 24, 2013 10:29 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> > >
> > > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> > >
> > > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > > --
> > >
> > > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > > I am trying to force the issue doing a lot of requests all at once.
> > > I reduced my MAX down to 1 to force contention. Everything seems
> > > to work pretty well. However, I did get the Connection Reset
> > > error twice on my machine! The logs were fairly regular looking
> > > since there is only one connection at a time. However, when this
> > > happened, it looks like it is sending 2 request at once. This
> > > shouldn't be happening because there is only one connection.
> > > (Granted, it is hard to tell if the logging is indicating a
> > > threading issue, or if the "Connection Reset" error caused the
> > > logging abnormality.)
> > >
> > > I don't need to block around the "httpClient.execute(request);" do I?
> > >
> > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > > --
> > >
> > > I do this to establish the connection:
> > >
> > > This is run once in the setup
> > > -----------------------------
> > > connectionManager = new
> > > PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > >
> > >
> > >
> > > This is called when I want to make a request
> > > --------------------------------------------
> > > response = httpClient.execute(request);
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 5:06 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Well, the user went all day without any errors while using the Java.net connector.
> > >
> > > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> > >
> > > Mark
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 1:47 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Oh, I forgot to add that all my requests are POST requests.
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 1:02 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> > >
> > > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> > >
> > > Mark
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 12:06 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> > >
> > > > It may well be that HttpURLConnection silently retries failed
> > > > requests
> > > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> > >
> > > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> > >
> > > public boolean isStale() {
> > > if (!isOpen()) {
> > > return true;
> > > }
> > > if (isEof()) {
> > > return true;
> > > }
> > > try {
> > > this.inbuffer.isDataAvailable(1);
> > > return isEof();
> > > } catch (SocketTimeoutException ex) {
> > > return false;
> > > } catch (IOException ex) {
> > > return true;
> > > }
> > > }
> > >
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 11:24 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > > >
> > >
> > > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> > >
> > >
> > > > Some other things of note:
> > > > - She is using JRE 6 Update 13. (There is a long story behind
> > > > this.)
> > > > - The application is launched via Webstart
> > > > - The SocketFactory is created with: fact = new
> > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory()
> > > > ,v
> > > > er
> > > > if
> > > > ier);
> > > >
> > >
> > > I do not thing any of these should matter.
> > >
> > > Oleg
> > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Friday, January 18, 2013 6:17 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > > Thanks for the explanation.
> > > > > > pro-actively evicting expired connections and connections
> > > > > > that have been idle
> > > > > Seems like just doing the stale check would more reliable, and
> > > > > speed does not seem to be a factory (especially with 4.2.3)
> > > > >
> > > > > I got in configured and distributed to this user and she is
> > > > > still having the same problem. Any other ideas? I double
> > > > > checked my code, and it looks like everything is correct.
> > > > > (She is not using a
> > > > > proxy.)
> > > > >
> > > >
> > > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > > >
> > > > Oleg
> > > >
> > > >
> > > > >
> > > > > Method that makes the request (from multiple threads)
> > > > > {
> > > > > response = httpClient.execute(request);
> > > > > }
> > > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > > {
> > > > > HttpParams params = new BasicHttpParams();
> > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > > configureProxy(params,protocol);
> > > > >
> > > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > > connectionManager.setMaxTotal(MAX);
> > > > >
> > > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > > }
> > > > > private void configureProxy(HttpParams params, String scheme) {
> > > > > Proxy proxy = getProxy();
> > > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > > //SSL or not, we set the proxy up as HTTP
> > > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > > }
> > > > > }
> > > > >
> > > > > }
> > > > >
> > > > > -----Original Message-----
> > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: Connection Reset errors
> > > > >
> > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > > >
> > > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > > >
> > > > > > I noticed that there is a stale connection check:
> > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_
> > > > > > CH
> > > > > > EC
> > > > > > K,
> > > > > > Boolean.TRUE);
> > > > > >
> > > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > > >
> > > > > > Mark
> > > > > >
> > > > >
> > > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > > To: 'HttpClient User Discussion'
> > > > > > Subject: Connection Reset errors
> > > > > >
> > > > > > I have a user getting a lot of Connection Reset errors. I
> > > > > > did not think this had do to with HttpClient, so much as
> > > > > > other factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > > >
> > > > > > This user is fine for a while, and then will get the
> > > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > > <Connector
> > > > > > port="5502"
> > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > > server="Unknown"
> > > > > > connectionLinger="0"
> > > > > > socket.soTimeout="300000"
> > > > > > SSLEnabled="true"
> > > > > > maxThreads="150"
> > > > > > scheme="https"
> > > > > > secure="true"
> > > > > > clientAuth="false"
> > > > > > keystoreFile="-----"
> > > > > > keystoreType="PKCS12"
> > > > > > keystorePass="-----"
> > > > > > sslProtocol="TLS"
> > > > > >
> > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_S
> > > > > > HA ,T LS _D HE
> > > > > > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_
> > > > > > 3D ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > > >
> > > > > >
> > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > > Message (Connection reset)
> > > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > > at
> > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > > ja
> > > > > > va:138)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > > >
> > > > > > ------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote:
> CLUE!
>
> I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
>
> I about doubled the size of the message I send, and it makes no difference.
>
> I put the iteration count in the HttpHeader so I could track it in the Tomcat access log. Tomcat shows that it receives this message (message 100) and that it thinks it succeeded (status == 200)
>
> The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
>
>
Tomcat by default closes persistent connections once it has been used to
serve 100 requests.
See 'maxKeepAliveRequests' parameter:
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Try setting this value to -1 and see if that makes the problem go away.
Oleg
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 2:01 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
>
> What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
>
> Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
> FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
> Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
> FINE: << " </BODY>[\n]"
> Thread-00, setSoTimeout(45000) called
>
> [snip]
>
> Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]: length = 176
>
> [snip]
>
> [Raw read]: length = 5
> 0000: 17 03 01 03 0A .....
> Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
> 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
> 0010: C8 08 ..
> Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception sending alert: java.net.SocketException: Connection reset by peer: socket write error Thread-00, called closeSocket() Thread-00, called close() Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Tuesday, January 29, 2013 12:39 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Don't know if this will help either...but here is hoping!
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Tuesday, January 29, 2013 9:03 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> > I ran my test program using 4.01 and it seems to fail even faster.
> > (Switching a few things around to get it to compile, or course. Using
> > ThreadSafeClientConnManager.)
> >
> > This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
> >
> > Still, the Java.net stuff is completely error free. (I can't say for
> > sure if she would notice automatic retries, if they occurred.)
> >
> > Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
> >
>
> Mark,
>
> This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
> So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
>
> So, running your application with SSL debugging turned on should be more
> informative:
>
> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
>
> Oleg
>
> > ------------------------------------
> > Stack trace and debug using 4.0.1
> >
> > FINE: Stale connection check
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.client.protocol.RequestAddCookies process
> > FINE: CookieSpec selected: best-match
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.client.DefaultRequestDirector execute
> > FINE: Attempt 1 to execute request
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013 12:08:45
> > PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "POST / HTTP/1.1[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Content-Type: binary[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Content-Length: 34[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Connection: Keep-Alive[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "[EOL]"
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> POST / HTTP/1.1
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Type: binary
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Length: 34
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> > sendRequestHeader
> > FINE: >> Connection: Keep-Alive
> > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection closed
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.client.DefaultRequestDirector execute
> > FINE: Closing the connection.
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection closed
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > FINE: Connection shut down
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> > releaseConnection
> > FINE: Released connection is not reusable.
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> > FINE: Releasing connection
> > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> > Thread-00:Jan 28, 2013 12:08:45 PM
> > org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> > FINE: Notifying no-one, there are no waiting threads
> > java.net.SocketException: Connection reset
> > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> > at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> > at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> > at java.lang.Thread.run(Thread.java:662)
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 12:30 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
> >
> > The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
> >
> > First call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > ManagedClientConnectionImpl.close:127
> > HttpRequestExecutor.closeConnection:144
> > HttpRequestExecutor.execute:131
> > DefaultRequestDirector.tryExecute:717
> > DefaultRequestDirector.execute:522
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> > Second call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > ManagedClientConnectionImpl.close:127
> > DefaultRequestDirector.tryExecute:723
> > DefaultRequestDirector.execute:522
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> > Third call to close()
> > ----------------------
> > DefaultClientConnection.close:168
> > HttpPoolEntry.close:89
> > AbstractConnPool.release:323
> > PoolingClientConnectionManager.releaseConnection:278
> > ManagedClientConnectionImpl.abortConnection:463
> > DefaultRequestDirector.abortConnection:1157
> > DefaultRequestDirector.execute:621
> > AbstractHttpClient.execute:906
> > AbstractHttpClient.execute:805
> > AbstractHttpClient.execute:784
> > TestNetwork$Worker.connect:187
> > TestNetwork$Worker.run:153
> > Thread.run:662
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:41 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
> >
> > Below is the last part of what I got.
> >
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection released: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > closeExpiredConnections
> > FINE: Closing expired connections
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > requestConnection
> > FINE: Connection request: [route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> > leaseConnection
> > FINE: Connection leased: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
> > FINE: Stale connection check
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.client.protocol.RequestAddCookies process
> > FINE: CookieSpec selected: best-match
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.client.protocol.RequestAuthCache process
> > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> > 11:38:01 AM
> > org.apache.http.client.protocol.RequestTargetAuthentication process
> > FINE: Target auth state: UNCHALLENGED
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.client.protocol.RequestProxyAuthentication process
> > FINE: Proxy auth state: UNCHALLENGED
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > FINE: Attempt 1 to execute request
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013 11:38:01
> > AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "POST / HTTP/1.1[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Content-Type: binary[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Content-Length: 34[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "Connection: Keep-Alive[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "[\r][\n]"
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> POST / HTTP/1.1
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Type: binary
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Content-Length: 34
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> > sendRequestHeader
> > FINE: >> Connection: Keep-Alive
> > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.client.DefaultRequestDirector tryExecute
> > FINE: Closing the connection.
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection shutdown
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.DefaultClientConnection close
> > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> > Thread-00:Jan 25, 2013 11:38:01 AM
> > org.apache.http.impl.conn.PoolingClientConnectionManager
> > releaseConnection
> > FINE: Connection released: [id: 0][route:
> > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> > allocated: 0 of 1; total allocated: 0 of 1]
> > java.net.SocketException: Connection reset
> > at java.net.SocketInputStream.read(SocketInputStream.java:168)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> > at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> > at java.lang.Thread.run(Thread.java:662)
> > Java Result: 1
> > BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:25 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 25, 2013 11:07 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Before the upgrade, she was using HttpClient 4.01.
> > ---
> > Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
> >
> > The website it is connecting to is just a Tomcat server running the
> > default ROOT webapp. It is delivering the default HTML page we have
> > set up on that. I would imagine that it is ignoring the POST and
> > treating it like a GET
> >
> > In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 25, 2013 7:46 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > Mark
> >
> > You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
> >
> > There are probably several things you might want to try out
> >
> > (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
> >
> > (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
> >
> > (3) Otherwise there is no way around getting very close and personal with Wireshark.
> >
> > Oleg
> >
> >
> >
> > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> > >
> > > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Thursday, January 24, 2013 10:29 AM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> > >
> > > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> > >
> > > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > > --
> > >
> > > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > > I am trying to force the issue doing a lot of requests all at once.
> > > I reduced my MAX down to 1 to force contention. Everything seems to
> > > work pretty well. However, I did get the Connection Reset error
> > > twice on my machine! The logs were fairly regular looking since
> > > there is only one connection at a time. However, when this
> > > happened, it looks like it is sending 2 request at once. This
> > > shouldn't be happening because there is only one connection.
> > > (Granted, it is hard to tell if the logging is indicating a
> > > threading issue, or if the "Connection Reset" error caused the
> > > logging abnormality.)
> > >
> > > I don't need to block around the "httpClient.execute(request);" do I?
> > >
> > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > > --
> > >
> > > I do this to establish the connection:
> > >
> > > This is run once in the setup
> > > -----------------------------
> > > connectionManager = new
> > > PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > >
> > >
> > >
> > > This is called when I want to make a request
> > > --------------------------------------------
> > > response = httpClient.execute(request);
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 5:06 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Well, the user went all day without any errors while using the Java.net connector.
> > >
> > > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> > >
> > > Mark
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 1:47 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Oh, I forgot to add that all my requests are POST requests.
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 1:02 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> > >
> > > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> > >
> > > Mark
> > >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Friday, January 18, 2013 12:06 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: RE: Connection Reset errors
> > >
> > > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> > >
> > > > It may well be that HttpURLConnection silently retries failed
> > > > requests
> > > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> > >
> > > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> > >
> > > public boolean isStale() {
> > > if (!isOpen()) {
> > > return true;
> > > }
> > > if (isEof()) {
> > > return true;
> > > }
> > > try {
> > > this.inbuffer.isDataAvailable(1);
> > > return isEof();
> > > } catch (SocketTimeoutException ex) {
> > > return false;
> > > } catch (IOException ex) {
> > > return true;
> > > }
> > > }
> > >
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 11:24 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > > >
> > >
> > > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> > >
> > >
> > > > Some other things of note:
> > > > - She is using JRE 6 Update 13. (There is a long story behind
> > > > this.)
> > > > - The application is launched via Webstart
> > > > - The SocketFactory is created with: fact = new
> > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),v
> > > > er
> > > > if
> > > > ier);
> > > >
> > >
> > > I do not thing any of these should matter.
> > >
> > > Oleg
> > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Friday, January 18, 2013 6:17 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > > Thanks for the explanation.
> > > > > > pro-actively evicting expired connections and connections that
> > > > > > have been idle
> > > > > Seems like just doing the stale check would more reliable, and
> > > > > speed does not seem to be a factory (especially with 4.2.3)
> > > > >
> > > > > I got in configured and distributed to this user and she is
> > > > > still having the same problem. Any other ideas? I double
> > > > > checked my code, and it looks like everything is correct. (She
> > > > > is not using a
> > > > > proxy.)
> > > > >
> > > >
> > > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > > >
> > > > Oleg
> > > >
> > > >
> > > > >
> > > > > Method that makes the request (from multiple threads)
> > > > > {
> > > > > response = httpClient.execute(request);
> > > > > }
> > > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > > {
> > > > > HttpParams params = new BasicHttpParams();
> > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > > configureProxy(params,protocol);
> > > > >
> > > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > > connectionManager.setMaxTotal(MAX);
> > > > >
> > > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > > }
> > > > > private void configureProxy(HttpParams params, String scheme) {
> > > > > Proxy proxy = getProxy();
> > > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > > //SSL or not, we set the proxy up as HTTP
> > > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > > }
> > > > > }
> > > > >
> > > > > }
> > > > >
> > > > > -----Original Message-----
> > > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > > To: HttpClient User Discussion
> > > > > Subject: Re: Connection Reset errors
> > > > >
> > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > > >
> > > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > > >
> > > > > > I noticed that there is a stale connection check:
> > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CH
> > > > > > EC
> > > > > > K,
> > > > > > Boolean.TRUE);
> > > > > >
> > > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > > >
> > > > > > Mark
> > > > > >
> > > > >
> > > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > > To: 'HttpClient User Discussion'
> > > > > > Subject: Connection Reset errors
> > > > > >
> > > > > > I have a user getting a lot of Connection Reset errors. I did
> > > > > > not think this had do to with HttpClient, so much as other
> > > > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > > >
> > > > > > This user is fine for a while, and then will get the
> > > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > > >
> > > > > > Thanks,
> > > > > > Mark
> > > > > >
> > > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > > <Connector
> > > > > > port="5502"
> > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > > server="Unknown"
> > > > > > connectionLinger="0"
> > > > > > socket.soTimeout="300000"
> > > > > > SSLEnabled="true"
> > > > > > maxThreads="150"
> > > > > > scheme="https"
> > > > > > secure="true"
> > > > > > clientAuth="false"
> > > > > > keystoreFile="-----"
> > > > > > keystoreType="PKCS12"
> > > > > > keystorePass="-----"
> > > > > > sslProtocol="TLS"
> > > > > >
> > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA
> > > > > > ,T LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3D
> > > > > > ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > > >
> > > > > >
> > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > > Message (Connection reset)
> > > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > > at
> > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > > ja
> > > > > > va:138)
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > --
> > > > > > --
> > > > > > --
> > > > > > - To unsubscribe, e-mail:
> > > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > > For additional commands, e-mail:
> > > > > > httpclient-users-help@hc.apache.org
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > ----------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
CLUE!
I opened up the cipher suites on Tomecat like I mentioned and then made my own socketfactory (limiting the cipher suites) as I saw in an example. Now when I run the program it dies at the same place every time...on the 101st connection. This has to be significant in some way...although I don't know how.
I about doubled the size of the message I send, and it makes no difference.
I put the iteration count in the HttpHeader so I could track it in the Tomcat access log. Tomcat shows that it receives this message (message 100) and that it thinks it succeeded (status == 200)
The SocketTimeoutException I noticed before is everywhere in the logs. This is from the StaleConnectionCheck and is likely irrelevant.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 2:01 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out Thread-00, setSoTimeout(45000) called
FINE: << " </BODY>[\n]"
Thread-00, setSoTimeout(45000) called
[snip]
Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]: length = 176
[snip]
[Raw read]: length = 5
0000: 17 03 01 03 0A .....
Thread-00, handling exception: java.net.SocketException: Connection reset %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext before ENCRYPTION: len = 18
0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
0010: C8 08 ..
Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception sending alert: java.net.SocketException: Connection reset by peer: socket write error Thread-00, called closeSocket() Thread-00, called close() Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 12:39 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Don't know if this will help either...but here is hoping!
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 9:03 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> I ran my test program using 4.01 and it seems to fail even faster.
> (Switching a few things around to get it to compile, or course. Using
> ThreadSafeClientConnManager.)
>
> This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
>
> Still, the Java.net stuff is completely error free. (I can't say for
> sure if she would notice automatic retries, if they occurred.)
>
> Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
>
Mark,
This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
So, running your application with SSL debugging turned on should be more
informative:
http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
Oleg
> ------------------------------------
> Stack trace and debug using 4.0.1
>
> FINE: Stale connection check
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013 12:08:45
> PM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Closing the connection.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection shut down
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> releaseConnection
> FINE: Released connection is not reusable.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> FINE: Releasing connection
> [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> FINE: Notifying no-one, there are no waiting threads
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> at java.lang.Thread.run(Thread.java:662)
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 12:30 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
>
> The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
>
> First call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> HttpRequestExecutor.closeConnection:144
> HttpRequestExecutor.execute:131
> DefaultRequestDirector.tryExecute:717
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Second call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> DefaultRequestDirector.tryExecute:723
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Third call to close()
> ----------------------
> DefaultClientConnection.close:168
> HttpPoolEntry.close:89
> AbstractConnPool.release:323
> PoolingClientConnectionManager.releaseConnection:278
> ManagedClientConnectionImpl.abortConnection:463
> DefaultRequestDirector.abortConnection:1157
> DefaultRequestDirector.execute:621
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:41 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
>
> Below is the last part of what I got.
>
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> closeExpiredConnections
> FINE: Closing expired connections
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> requestConnection
> FINE: Connection request: [route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> leaseConnection
> FINE: Connection leased: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Stale connection check
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAuthCache process
> FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> 11:38:01 AM
> org.apache.http.client.protocol.RequestTargetAuthentication process
> FINE: Target auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestProxyAuthentication process
> FINE: Proxy auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013 11:38:01
> AM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Closing the connection.
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 0 of 1; total allocated: 0 of 1]
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> at java.lang.Thread.run(Thread.java:662)
> Java Result: 1
> BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:25 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:07 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Before the upgrade, she was using HttpClient 4.01.
> ---
> Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
>
> The website it is connecting to is just a Tomcat server running the
> default ROOT webapp. It is delivering the default HTML page we have
> set up on that. I would imagine that it is ignoring the POST and
> treating it like a GET
>
> In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 25, 2013 7:46 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> Mark
>
> You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
>
> There are probably several things you might want to try out
>
> (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
>
> (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
>
> (3) Otherwise there is no way around getting very close and personal with Wireshark.
>
> Oleg
>
>
>
> On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> >
> > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Thursday, January 24, 2013 10:29 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> >
> > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> >
> > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > --
> >
> > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > I am trying to force the issue doing a lot of requests all at once.
> > I reduced my MAX down to 1 to force contention. Everything seems to
> > work pretty well. However, I did get the Connection Reset error
> > twice on my machine! The logs were fairly regular looking since
> > there is only one connection at a time. However, when this
> > happened, it looks like it is sending 2 request at once. This
> > shouldn't be happening because there is only one connection.
> > (Granted, it is hard to tell if the logging is indicating a
> > threading issue, or if the "Connection Reset" error caused the
> > logging abnormality.)
> >
> > I don't need to block around the "httpClient.execute(request);" do I?
> >
> > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > --
> >
> > I do this to establish the connection:
> >
> > This is run once in the setup
> > -----------------------------
> > connectionManager = new
> > PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> >
> >
> >
> > This is called when I want to make a request
> > --------------------------------------------
> > response = httpClient.execute(request);
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 5:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Well, the user went all day without any errors while using the Java.net connector.
> >
> > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> >
> > Mark
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:47 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Oh, I forgot to add that all my requests are POST requests.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:02 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> >
> > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> >
> > Mark
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 12:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> >
> > > It may well be that HttpURLConnection silently retries failed
> > > requests
> > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> >
> > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> >
> > public boolean isStale() {
> > if (!isOpen()) {
> > return true;
> > }
> > if (isEof()) {
> > return true;
> > }
> > try {
> > this.inbuffer.isDataAvailable(1);
> > return isEof();
> > } catch (SocketTimeoutException ex) {
> > return false;
> > } catch (IOException ex) {
> > return true;
> > }
> > }
> >
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 11:24 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > >
> >
> > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> >
> >
> > > Some other things of note:
> > > - She is using JRE 6 Update 13. (There is a long story behind
> > > this.)
> > > - The application is launched via Webstart
> > > - The SocketFactory is created with: fact = new
> > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),v
> > > er
> > > if
> > > ier);
> > >
> >
> > I do not thing any of these should matter.
> >
> > Oleg
> >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 6:17 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > Thanks for the explanation.
> > > > > pro-actively evicting expired connections and connections that
> > > > > have been idle
> > > > Seems like just doing the stale check would more reliable, and
> > > > speed does not seem to be a factory (especially with 4.2.3)
> > > >
> > > > I got in configured and distributed to this user and she is
> > > > still having the same problem. Any other ideas? I double
> > > > checked my code, and it looks like everything is correct. (She
> > > > is not using a
> > > > proxy.)
> > > >
> > >
> > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > >
> > > Oleg
> > >
> > >
> > > >
> > > > Method that makes the request (from multiple threads)
> > > > {
> > > > response = httpClient.execute(request);
> > > > }
> > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > {
> > > > HttpParams params = new BasicHttpParams();
> > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > configureProxy(params,protocol);
> > > >
> > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > }
> > > > private void configureProxy(HttpParams params, String scheme) {
> > > > Proxy proxy = getProxy();
> > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > //SSL or not, we set the proxy up as HTTP
> > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > }
> > > > }
> > > >
> > > > }
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > >
> > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > >
> > > > > I noticed that there is a stale connection check:
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CH
> > > > > EC
> > > > > K,
> > > > > Boolean.TRUE);
> > > > >
> > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > >
> > > > > Mark
> > > > >
> > > >
> > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > To: 'HttpClient User Discussion'
> > > > > Subject: Connection Reset errors
> > > > >
> > > > > I have a user getting a lot of Connection Reset errors. I did
> > > > > not think this had do to with HttpClient, so much as other
> > > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > >
> > > > > This user is fine for a while, and then will get the
> > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > <Connector
> > > > > port="5502"
> > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > server="Unknown"
> > > > > connectionLinger="0"
> > > > > socket.soTimeout="300000"
> > > > > SSLEnabled="true"
> > > > > maxThreads="150"
> > > > > scheme="https"
> > > > > secure="true"
> > > > > clientAuth="false"
> > > > > keystoreFile="-----"
> > > > > keystoreType="PKCS12"
> > > > > keystorePass="-----"
> > > > > sslProtocol="TLS"
> > > > >
> > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA
> > > > > ,T LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3D
> > > > > ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > >
> > > > >
> > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > Message (Connection reset)
> > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > at
> > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > ja
> > > > > va:138)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
I saw some things on the web with this error, but those seemed to be errors on startup because particular protocols where not enabled. I changed my Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites attribute. Similar result. However I saw the below in the debug output.
What is the socket timeout doing there, and why is there a SocketTimeoutException when this is all occurring in the same second.
Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
FINE: << " <p>If you page is not redirected, click <a href="JViewer/">here</a>[\n]"
Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire
Thread-00, handling exception: java.net.SocketTimeoutException: Read timed out
Thread-00, setSoTimeout(45000) called
FINE: << " </BODY>[\n]"
Thread-00, setSoTimeout(45000) called
[snip]
Thread-00, WRITE: TLSv1 Application Data, length = 171
[Raw write]: length = 176
[snip]
[Raw read]: length = 5
0000: 17 03 01 03 0A .....
Thread-00, handling exception: java.net.SocketException: Connection reset
%% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5]
Thread-00, SEND TLSv1 ALERT: fatal, description = unexpected_message
Padded plaintext before ENCRYPTION: len = 18
0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B
0010: C8 08 ..
Thread-00, WRITE: TLSv1 Alert, length = 18
Thread-00, Exception sending alert: java.net.SocketException: Connection reset by peer: socket write error
Thread-00, called closeSocket()
Thread-00, called close()
Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Tuesday, January 29, 2013 12:39 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Don't know if this will help either...but here is hoping!
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 9:03 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> I ran my test program using 4.01 and it seems to fail even faster.
> (Switching a few things around to get it to compile, or course. Using
> ThreadSafeClientConnManager.)
>
> This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
>
> Still, the Java.net stuff is completely error free. (I can't say for
> sure if she would notice automatic retries, if they occurred.)
>
> Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
>
Mark,
This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
So, running your application with SSL debugging turned on should be more
informative:
http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
Oleg
> ------------------------------------
> Stack trace and debug using 4.0.1
>
> FINE: Stale connection check
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013 12:08:45
> PM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Closing the connection.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection shut down
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> releaseConnection
> FINE: Released connection is not reusable.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> FINE: Releasing connection
> [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> FINE: Notifying no-one, there are no waiting threads
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> at java.lang.Thread.run(Thread.java:662)
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 12:30 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
>
> The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
>
> First call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> HttpRequestExecutor.closeConnection:144
> HttpRequestExecutor.execute:131
> DefaultRequestDirector.tryExecute:717
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Second call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> DefaultRequestDirector.tryExecute:723
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Third call to close()
> ----------------------
> DefaultClientConnection.close:168
> HttpPoolEntry.close:89
> AbstractConnPool.release:323
> PoolingClientConnectionManager.releaseConnection:278
> ManagedClientConnectionImpl.abortConnection:463
> DefaultRequestDirector.abortConnection:1157
> DefaultRequestDirector.execute:621
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:41 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
>
> Below is the last part of what I got.
>
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> closeExpiredConnections
> FINE: Closing expired connections
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> requestConnection
> FINE: Connection request: [route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> leaseConnection
> FINE: Connection leased: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Stale connection check
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAuthCache process
> FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> 11:38:01 AM
> org.apache.http.client.protocol.RequestTargetAuthentication process
> FINE: Target auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestProxyAuthentication process
> FINE: Proxy auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013 11:38:01
> AM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Closing the connection.
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 0 of 1; total allocated: 0 of 1]
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> at java.lang.Thread.run(Thread.java:662)
> Java Result: 1
> BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:25 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:07 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Before the upgrade, she was using HttpClient 4.01.
> ---
> Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
>
> The website it is connecting to is just a Tomcat server running the
> default ROOT webapp. It is delivering the default HTML page we have
> set up on that. I would imagine that it is ignoring the POST and
> treating it like a GET
>
> In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 25, 2013 7:46 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> Mark
>
> You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
>
> There are probably several things you might want to try out
>
> (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
>
> (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
>
> (3) Otherwise there is no way around getting very close and personal with Wireshark.
>
> Oleg
>
>
>
> On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> >
> > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Thursday, January 24, 2013 10:29 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> >
> > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> >
> > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > --
> >
> > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > I am trying to force the issue doing a lot of requests all at once.
> > I reduced my MAX down to 1 to force contention. Everything seems to
> > work pretty well. However, I did get the Connection Reset error
> > twice on my machine! The logs were fairly regular looking since
> > there is only one connection at a time. However, when this
> > happened, it looks like it is sending 2 request at once. This
> > shouldn't be happening because there is only one connection.
> > (Granted, it is hard to tell if the logging is indicating a
> > threading issue, or if the "Connection Reset" error caused the
> > logging abnormality.)
> >
> > I don't need to block around the "httpClient.execute(request);" do I?
> >
> > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > --
> >
> > I do this to establish the connection:
> >
> > This is run once in the setup
> > -----------------------------
> > connectionManager = new
> > PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> >
> >
> >
> > This is called when I want to make a request
> > --------------------------------------------
> > response = httpClient.execute(request);
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 5:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Well, the user went all day without any errors while using the Java.net connector.
> >
> > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> >
> > Mark
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:47 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Oh, I forgot to add that all my requests are POST requests.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:02 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> >
> > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> >
> > Mark
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 12:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> >
> > > It may well be that HttpURLConnection silently retries failed
> > > requests
> > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> >
> > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> >
> > public boolean isStale() {
> > if (!isOpen()) {
> > return true;
> > }
> > if (isEof()) {
> > return true;
> > }
> > try {
> > this.inbuffer.isDataAvailable(1);
> > return isEof();
> > } catch (SocketTimeoutException ex) {
> > return false;
> > } catch (IOException ex) {
> > return true;
> > }
> > }
> >
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 11:24 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > >
> >
> > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> >
> >
> > > Some other things of note:
> > > - She is using JRE 6 Update 13. (There is a long story behind
> > > this.)
> > > - The application is launched via Webstart
> > > - The SocketFactory is created with: fact = new
> > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),v
> > > er
> > > if
> > > ier);
> > >
> >
> > I do not thing any of these should matter.
> >
> > Oleg
> >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 6:17 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > Thanks for the explanation.
> > > > > pro-actively evicting expired connections and connections that
> > > > > have been idle
> > > > Seems like just doing the stale check would more reliable, and
> > > > speed does not seem to be a factory (especially with 4.2.3)
> > > >
> > > > I got in configured and distributed to this user and she is
> > > > still having the same problem. Any other ideas? I double
> > > > checked my code, and it looks like everything is correct. (She
> > > > is not using a
> > > > proxy.)
> > > >
> > >
> > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > >
> > > Oleg
> > >
> > >
> > > >
> > > > Method that makes the request (from multiple threads)
> > > > {
> > > > response = httpClient.execute(request);
> > > > }
> > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > {
> > > > HttpParams params = new BasicHttpParams();
> > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > configureProxy(params,protocol);
> > > >
> > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > }
> > > > private void configureProxy(HttpParams params, String scheme) {
> > > > Proxy proxy = getProxy();
> > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > //SSL or not, we set the proxy up as HTTP
> > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > }
> > > > }
> > > >
> > > > }
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > >
> > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > >
> > > > > I noticed that there is a stale connection check:
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CH
> > > > > EC
> > > > > K,
> > > > > Boolean.TRUE);
> > > > >
> > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > >
> > > > > Mark
> > > > >
> > > >
> > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > To: 'HttpClient User Discussion'
> > > > > Subject: Connection Reset errors
> > > > >
> > > > > I have a user getting a lot of Connection Reset errors. I did
> > > > > not think this had do to with HttpClient, so much as other
> > > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > >
> > > > > This user is fine for a while, and then will get the
> > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > <Connector
> > > > > port="5502"
> > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > server="Unknown"
> > > > > connectionLinger="0"
> > > > > socket.soTimeout="300000"
> > > > > SSLEnabled="true"
> > > > > maxThreads="150"
> > > > > scheme="https"
> > > > > secure="true"
> > > > > clientAuth="false"
> > > > > keystoreFile="-----"
> > > > > keystoreType="PKCS12"
> > > > > keystorePass="-----"
> > > > > sslProtocol="TLS"
> > > > >
> > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA
> > > > > ,T LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3D
> > > > > ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > >
> > > > >
> > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > Message (Connection reset)
> > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > at
> > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > ja
> > > > > va:138)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Don't know if this will help either...but here is hoping!
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Tuesday, January 29, 2013 9:03 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> I ran my test program using 4.01 and it seems to fail even faster.
> (Switching a few things around to get it to compile, or course. Using
> ThreadSafeClientConnManager.)
>
> This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
>
> Still, the Java.net stuff is completely error free. (I can't say for
> sure if she would notice automatic retries, if they occurred.)
>
> Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
>
Mark,
This log unfortunately does not help. Interesting bit, though, it that a connection reset happens while trying to read a head of a new message.
So, clearly the stale connection check did not work. In the former days (Java 1.2 and possibly 1.3) state connection checking was simply useless with SSL encrypted connections. Apparently it can still be ineffective even with modern JREs.
So, running your application with SSL debugging turned on should be more
informative:
http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
Oleg
> ------------------------------------
> Stack trace and debug using 4.0.1
>
> FINE: Stale connection check
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013 12:08:45
> PM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013
> 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Closing the connection.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection shut down
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager
> releaseConnection
> FINE: Released connection is not reusable.
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> FINE: Releasing connection
> [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> Thread-00:Jan 28, 2013 12:08:45 PM
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> FINE: Notifying no-one, there are no waiting threads
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> at java.lang.Thread.run(Thread.java:662)
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 12:30 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
>
> The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
>
> First call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> HttpRequestExecutor.closeConnection:144
> HttpRequestExecutor.execute:131
> DefaultRequestDirector.tryExecute:717
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Second call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> DefaultRequestDirector.tryExecute:723
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Third call to close()
> ----------------------
> DefaultClientConnection.close:168
> HttpPoolEntry.close:89
> AbstractConnPool.release:323
> PoolingClientConnectionManager.releaseConnection:278
> ManagedClientConnectionImpl.abortConnection:463
> DefaultRequestDirector.abortConnection:1157
> DefaultRequestDirector.execute:621
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:41 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
>
> Below is the last part of what I got.
>
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502] can be kept alive
> indefinitely Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> closeExpiredConnections
> FINE: Closing expired connections
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> requestConnection
> FINE: Connection request: [route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager
> leaseConnection
> FINE: Connection leased: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Stale connection check
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestAuthCache process
> FINE: Auth cache not set in the context Thread-00:Jan 25, 2013
> 11:38:01 AM
> org.apache.http.client.protocol.RequestTargetAuthentication process
> FINE: Target auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.client.protocol.RequestProxyAuthentication process
> FINE: Proxy auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013 11:38:01
> AM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013
> 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection
> sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Closing the connection.
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
> Thread-00:Jan 25, 2013 11:38:01 AM
> org.apache.http.impl.conn.PoolingClientConnectionManager
> releaseConnection
> FINE: Connection released: [id: 0][route:
> {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route
> allocated: 0 of 1; total allocated: 0 of 1]
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> at java.lang.Thread.run(Thread.java:662)
> Java Result: 1
> BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:25 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:07 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Before the upgrade, she was using HttpClient 4.01.
> ---
> Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
>
> The website it is connecting to is just a Tomcat server running the
> default ROOT webapp. It is delivering the default HTML page we have
> set up on that. I would imagine that it is ignoring the POST and
> treating it like a GET
>
> In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 25, 2013 7:46 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> Mark
>
> You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
>
> There are probably several things you might want to try out
>
> (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
>
> (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
>
> (3) Otherwise there is no way around getting very close and personal with Wireshark.
>
> Oleg
>
>
>
> On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> >
> > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Thursday, January 24, 2013 10:29 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> >
> > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> >
> > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > --
> >
> > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > I am trying to force the issue doing a lot of requests all at once.
> > I reduced my MAX down to 1 to force contention. Everything seems to
> > work pretty well. However, I did get the Connection Reset error
> > twice on my machine! The logs were fairly regular looking since
> > there is only one connection at a time. However, when this
> > happened, it looks like it is sending 2 request at once. This
> > shouldn't be happening because there is only one connection.
> > (Granted, it is hard to tell if the logging is indicating a
> > threading issue, or if the "Connection Reset" error caused the
> > logging abnormality.)
> >
> > I don't need to block around the "httpClient.execute(request);" do I?
> >
> > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > --
> >
> > I do this to establish the connection:
> >
> > This is run once in the setup
> > -----------------------------
> > connectionManager = new
> > PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> >
> >
> >
> > This is called when I want to make a request
> > --------------------------------------------
> > response = httpClient.execute(request);
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 5:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Well, the user went all day without any errors while using the Java.net connector.
> >
> > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> >
> > Mark
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:47 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Oh, I forgot to add that all my requests are POST requests.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:02 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> >
> > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> >
> > Mark
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 12:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> >
> > > It may well be that HttpURLConnection silently retries failed
> > > requests
> > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> >
> > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> >
> > public boolean isStale() {
> > if (!isOpen()) {
> > return true;
> > }
> > if (isEof()) {
> > return true;
> > }
> > try {
> > this.inbuffer.isDataAvailable(1);
> > return isEof();
> > } catch (SocketTimeoutException ex) {
> > return false;
> > } catch (IOException ex) {
> > return true;
> > }
> > }
> >
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 11:24 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > >
> >
> > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> >
> >
> > > Some other things of note:
> > > - She is using JRE 6 Update 13. (There is a long story behind
> > > this.)
> > > - The application is launched via Webstart
> > > - The SocketFactory is created with: fact = new
> > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),v
> > > er
> > > if
> > > ier);
> > >
> >
> > I do not thing any of these should matter.
> >
> > Oleg
> >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 6:17 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > Thanks for the explanation.
> > > > > pro-actively evicting expired connections and connections that
> > > > > have been idle
> > > > Seems like just doing the stale check would more reliable, and
> > > > speed does not seem to be a factory (especially with 4.2.3)
> > > >
> > > > I got in configured and distributed to this user and she is
> > > > still having the same problem. Any other ideas? I double
> > > > checked my code, and it looks like everything is correct. (She
> > > > is not using a
> > > > proxy.)
> > > >
> > >
> > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > >
> > > Oleg
> > >
> > >
> > > >
> > > > Method that makes the request (from multiple threads)
> > > > {
> > > > response = httpClient.execute(request);
> > > > }
> > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > {
> > > > HttpParams params = new BasicHttpParams();
> > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > configureProxy(params,protocol);
> > > >
> > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > }
> > > > private void configureProxy(HttpParams params, String scheme) {
> > > > Proxy proxy = getProxy();
> > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > //SSL or not, we set the proxy up as HTTP
> > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > }
> > > > }
> > > >
> > > > }
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > >
> > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > >
> > > > > I noticed that there is a stale connection check:
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CH
> > > > > EC
> > > > > K,
> > > > > Boolean.TRUE);
> > > > >
> > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > >
> > > > > Mark
> > > > >
> > > >
> > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > To: 'HttpClient User Discussion'
> > > > > Subject: Connection Reset errors
> > > > >
> > > > > I have a user getting a lot of Connection Reset errors. I did
> > > > > not think this had do to with HttpClient, so much as other
> > > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > >
> > > > > This user is fine for a while, and then will get the
> > > > > Connection Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > <Connector
> > > > > port="5502"
> > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > server="Unknown"
> > > > > connectionLinger="0"
> > > > > socket.soTimeout="300000"
> > > > > SSLEnabled="true"
> > > > > maxThreads="150"
> > > > > scheme="https"
> > > > > secure="true"
> > > > > clientAuth="false"
> > > > > keystoreFile="-----"
> > > > > keystoreType="PKCS12"
> > > > > keystorePass="-----"
> > > > > sslProtocol="TLS"
> > > > >
> > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA
> > > > > ,T LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3D
> > > > > ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > >
> > > > >
> > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > Message (Connection reset)
> > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > at
> > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > ja
> > > > > va:138)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > --------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote:
> I ran my test program using 4.01 and it seems to fail even faster. (Switching a few things around to get it to compile, or course. Using ThreadSafeClientConnManager.)
>
> This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
>
> Still, the Java.net stuff is completely error free. (I can't say for sure if she would notice automatic retries, if they occurred.)
>
> Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
>
Mark,
This log unfortunately does not help. Interesting bit, though, it that a
connection reset happens while trying to read a head of a new message.
So, clearly the stale connection check did not work. In the former days
(Java 1.2 and possibly 1.3) state connection checking was simply useless
with SSL encrypted connections. Apparently it can still be ineffective
even with modern JREs.
So, running your application with SSL debugging turned on should be more
informative:
http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
Oleg
> ------------------------------------
> Stack trace and debug using 4.0.1
>
> FINE: Stale connection check
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "[EOL]"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Closing the connection.
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection closed
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection shut down
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager releaseConnection
> FINE: Released connection is not reusable.
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
> FINE: Releasing connection [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
> Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
> FINE: Notifying no-one, there are no waiting threads
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
> at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
> at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
> at java.lang.Thread.run(Thread.java:662)
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 12:30 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
>
> The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
>
> First call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> HttpRequestExecutor.closeConnection:144
> HttpRequestExecutor.execute:131
> DefaultRequestDirector.tryExecute:717
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Second call to close()
> ----------------------
> DefaultClientConnection.close:168
> ManagedClientConnectionImpl.close:127
> DefaultRequestDirector.tryExecute:723
> DefaultRequestDirector.execute:522
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
> Third call to close()
> ----------------------
> DefaultClientConnection.close:168
> HttpPoolEntry.close:89
> AbstractConnPool.release:323
> PoolingClientConnectionManager.releaseConnection:278
> ManagedClientConnectionImpl.abortConnection:463
> DefaultRequestDirector.abortConnection:1157
> DefaultRequestDirector.execute:621
> AbstractHttpClient.execute:906
> AbstractHttpClient.execute:805
> AbstractHttpClient.execute:784
> TestNetwork$Worker.connect:187
> TestNetwork$Worker.run:153
> Thread.run:662
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:41 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
>
> Below is the last part of what I got.
>
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
> FINE: Connection [id: 0][route: {s}->https://x86test.dev.donnell.com:5502] can be kept alive indefinitely Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
> FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager closeExpiredConnections
> FINE: Closing expired connections
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager requestConnection
> FINE: Connection request: [route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager leaseConnection
> FINE: Connection leased: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
> FINE: Stale connection check
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAddCookies process
> FINE: CookieSpec selected: best-match
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process
> FINE: Auth cache not set in the context
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestTargetAuthentication process
> FINE: Target auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestProxyAuthentication process
> FINE: Proxy auth state: UNCHALLENGED
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Attempt 1 to execute request
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: Sending request: POST / HTTP/1.1
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "POST / HTTP/1.1[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Type: binary[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Content-Length: 34[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "Connection: Keep-Alive[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "[\r][\n]"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> POST / HTTP/1.1
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Type: binary
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Content-Length: 34
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
> FINE: >> Connection: Keep-Alive
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
> FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
> FINE: Closing the connection.
> Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection shutdown
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
> FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
> FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 0 of 1; total allocated: 0 of 1]
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:168)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
> at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
> at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
> at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
> at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
> at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
> at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
> at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
> at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
> at java.lang.Thread.run(Thread.java:662)
> Java Result: 1
> BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:25 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 25, 2013 11:07 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Before the upgrade, she was using HttpClient 4.01.
> ---
> Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
>
> The website it is connecting to is just a Tomcat server running the default ROOT webapp. It is delivering the default HTML page we have set up on that. I would imagine that it is ignoring the POST and treating it like a GET
>
> In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 25, 2013 7:46 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> Mark
>
> You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
>
> There are probably several things you might want to try out
>
> (1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
>
> (2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
>
> (3) Otherwise there is no way around getting very close and personal with Wireshark.
>
> Oleg
>
>
>
> On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> > Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
> >
> > Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Thursday, January 24, 2013 10:29 AM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
> >
> > So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
> >
> > The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> > --
> >
> > Another avenue I am investigating is trying to force the issue by making the connections work harder.
> > I am trying to force the issue doing a lot of requests all at once. I
> > reduced my MAX down to 1 to force contention. Everything seems to
> > work pretty well. However, I did get the Connection Reset error twice
> > on my machine! The logs were fairly regular looking since there is
> > only one connection at a time. However, when this happened, it looks
> > like it is sending 2 request at once. This shouldn't be happening
> > because there is only one connection. (Granted, it is hard to tell if
> > the logging is indicating a threading issue, or if the "Connection
> > Reset" error caused the logging abnormality.)
> >
> > I don't need to block around the "httpClient.execute(request);" do I?
> >
> > I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> > --
> >
> > I do this to establish the connection:
> >
> > This is run once in the setup
> > -----------------------------
> > connectionManager = new
> > PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> >
> >
> >
> > This is called when I want to make a request
> > --------------------------------------------
> > response = httpClient.execute(request);
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 5:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Well, the user went all day without any errors while using the Java.net connector.
> >
> > The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
> >
> > Mark
> >
> >
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:47 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Oh, I forgot to add that all my requests are POST requests.
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 1:02 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
> >
> > However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
> >
> > Mark
> >
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Friday, January 18, 2013 12:06 PM
> > To: 'HttpClient User Discussion'
> > Subject: RE: Connection Reset errors
> >
> > Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> >
> > > It may well be that HttpURLConnection silently retries failed
> > > requests
> > Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
> >
> > She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
> >
> > public boolean isStale() {
> > if (!isOpen()) {
> > return true;
> > }
> > if (isEof()) {
> > return true;
> > }
> > try {
> > this.inbuffer.isDataAvailable(1);
> > return isEof();
> > } catch (SocketTimeoutException ex) {
> > return false;
> > } catch (IOException ex) {
> > return true;
> > }
> > }
> >
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 11:24 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> > >
> >
> > It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> >
> >
> > > Some other things of note:
> > > - She is using JRE 6 Update 13. (There is a long story behind
> > > this.)
> > > - The application is launched via Webstart
> > > - The SocketFactory is created with: fact = new
> > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),ver
> > > if
> > > ier);
> > >
> >
> > I do not thing any of these should matter.
> >
> > Oleg
> >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Friday, January 18, 2013 6:17 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > > Thanks for the explanation.
> > > > > pro-actively evicting expired connections and connections that
> > > > > have been idle
> > > > Seems like just doing the stale check would more reliable, and
> > > > speed does not seem to be a factory (especially with 4.2.3)
> > > >
> > > > I got in configured and distributed to this user and she is still
> > > > having the same problem. Any other ideas? I double checked my
> > > > code, and it looks like everything is correct. (She is not using
> > > > a
> > > > proxy.)
> > > >
> > >
> > > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> > >
> > > Oleg
> > >
> > >
> > > >
> > > > Method that makes the request (from multiple threads)
> > > > {
> > > > response = httpClient.execute(request);
> > > > }
> > > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > > {
> > > > HttpParams params = new BasicHttpParams();
> > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > > configureProxy(params,protocol);
> > > >
> > > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > > connectionManager.setMaxTotal(MAX);
> > > >
> > > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > > }
> > > > private void configureProxy(HttpParams params, String scheme) {
> > > > Proxy proxy = getProxy();
> > > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > > //SSL or not, we set the proxy up as HTTP
> > > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > > if (!scheme.equals(SCHEME_HTTP)) {
> > > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > > }
> > > > }
> > > >
> > > > }
> > > >
> > > > -----Original Message-----
> > > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > > Sent: Thursday, January 17, 2013 7:21 AM
> > > > To: HttpClient User Discussion
> > > > Subject: Re: Connection Reset errors
> > > >
> > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > > >
> > > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > > >
> > > > > I noticed that there is a stale connection check:
> > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHEC
> > > > > K,
> > > > > Boolean.TRUE);
> > > > >
> > > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > > >
> > > > > Mark
> > > > >
> > > >
> > > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > > To: 'HttpClient User Discussion'
> > > > > Subject: Connection Reset errors
> > > > >
> > > > > I have a user getting a lot of Connection Reset errors. I did
> > > > > not think this had do to with HttpClient, so much as other
> > > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > > >
> > > > > This user is fine for a while, and then will get the Connection
> > > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > > >
> > > > > Thanks,
> > > > > Mark
> > > > >
> > > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > > <Connector
> > > > > port="5502"
> > > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > > server="Unknown"
> > > > > connectionLinger="0"
> > > > > socket.soTimeout="300000"
> > > > > SSLEnabled="true"
> > > > > maxThreads="150"
> > > > > scheme="https"
> > > > > secure="true"
> > > > > clientAuth="false"
> > > > > keystoreFile="-----"
> > > > > keystoreType="PKCS12"
> > > > > keystorePass="-----"
> > > > > sslProtocol="TLS"
> > > > >
> > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,T
> > > > > LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES
> > > > > _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > > >
> > > > >
> > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > > Message (Connection reset)
> > > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > > at
> > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > > ja
> > > > > va:138)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > > > ----------------------------------------------------------------
> > > > > --
> > > > > --
> > > > > - To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
I ran my test program using 4.01 and it seems to fail even faster. (Switching a few things around to get it to compile, or course. Using ThreadSafeClientConnManager.)
This is odd, since the user said she didn't experience the problem until the upgrade which gave her the newer version. Must also be something else going on.
Still, the Java.net stuff is completely error free. (I can't say for sure if she would notice automatic retries, if they occurred.)
Is there any way I can reduce the occurrences of the error? Something I could set for specific users that are experiencing this problem?
------------------------------------
Stack trace and debug using 4.0.1
FINE: Stale connection check
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.client.protocol.RequestAddCookies process
FINE: CookieSpec selected: best-match
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector execute
FINE: Attempt 1 to execute request
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: Sending request: POST / HTTP/1.1
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
FINE: >> "POST / HTTP/1.1[EOL]"
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Type: binary[EOL]"
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Length: 34[EOL]"
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]"
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
FINE: >> "Connection: Keep-Alive[EOL]"
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
FINE: >> "[EOL]"
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> POST / HTTP/1.1
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Type: binary
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Length: 34
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Host: x86test.dev.donnell.com:5502
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Connection: Keep-Alive
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire
FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX"
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection closed
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector execute
FINE: Closing the connection.
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection closed
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection shutdown
FINE: Connection shut down
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager releaseConnection
FINE: Released connection is not reusable.
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry
FINE: Releasing connection [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null]
Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread
FINE: Notifying no-one, there are no waiting threads
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233)
at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100)
at org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227)
at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191)
at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157)
at java.lang.Thread.run(Thread.java:662)
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 12:30 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
First call to close()
----------------------
DefaultClientConnection.close:168
ManagedClientConnectionImpl.close:127
HttpRequestExecutor.closeConnection:144
HttpRequestExecutor.execute:131
DefaultRequestDirector.tryExecute:717
DefaultRequestDirector.execute:522
AbstractHttpClient.execute:906
AbstractHttpClient.execute:805
AbstractHttpClient.execute:784
TestNetwork$Worker.connect:187
TestNetwork$Worker.run:153
Thread.run:662
Second call to close()
----------------------
DefaultClientConnection.close:168
ManagedClientConnectionImpl.close:127
DefaultRequestDirector.tryExecute:723
DefaultRequestDirector.execute:522
AbstractHttpClient.execute:906
AbstractHttpClient.execute:805
AbstractHttpClient.execute:784
TestNetwork$Worker.connect:187
TestNetwork$Worker.run:153
Thread.run:662
Third call to close()
----------------------
DefaultClientConnection.close:168
HttpPoolEntry.close:89
AbstractConnPool.release:323
PoolingClientConnectionManager.releaseConnection:278
ManagedClientConnectionImpl.abortConnection:463
DefaultRequestDirector.abortConnection:1157
DefaultRequestDirector.execute:621
AbstractHttpClient.execute:906
AbstractHttpClient.execute:805
AbstractHttpClient.execute:784
TestNetwork$Worker.connect:187
TestNetwork$Worker.run:153
Thread.run:662
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:41 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
Below is the last part of what I got.
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection [id: 0][route: {s}->https://x86test.dev.donnell.com:5502] can be kept alive indefinitely Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager closeExpiredConnections
FINE: Closing expired connections
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager requestConnection
FINE: Connection request: [route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager leaseConnection
FINE: Connection leased: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
FINE: Stale connection check
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAddCookies process
FINE: CookieSpec selected: best-match
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process
FINE: Auth cache not set in the context
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestTargetAuthentication process
FINE: Target auth state: UNCHALLENGED
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestProxyAuthentication process
FINE: Proxy auth state: UNCHALLENGED
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
FINE: Attempt 1 to execute request
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: Sending request: POST / HTTP/1.1
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "POST / HTTP/1.1[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Type: binary[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Length: 34[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Connection: Keep-Alive[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> POST / HTTP/1.1
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Type: binary
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Length: 34
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Connection: Keep-Alive
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
FINE: Closing the connection.
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection shutdown
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 0 of 1; total allocated: 0 of 1]
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
at java.lang.Thread.run(Thread.java:662)
Java Result: 1
BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:25 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:07 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Before the upgrade, she was using HttpClient 4.01.
---
Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
The website it is connecting to is just a Tomcat server running the default ROOT webapp. It is delivering the default HTML page we have set up on that. I would imagine that it is ignoring the POST and treating it like a GET
In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 25, 2013 7:46 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
Mark
You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
There are probably several things you might want to try out
(1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
(2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
(3) Otherwise there is no way around getting very close and personal with Wireshark.
Oleg
On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
>
> Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Thursday, January 24, 2013 10:29 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
>
> So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
>
> The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> --
>
> Another avenue I am investigating is trying to force the issue by making the connections work harder.
> I am trying to force the issue doing a lot of requests all at once. I
> reduced my MAX down to 1 to force contention. Everything seems to
> work pretty well. However, I did get the Connection Reset error twice
> on my machine! The logs were fairly regular looking since there is
> only one connection at a time. However, when this happened, it looks
> like it is sending 2 request at once. This shouldn't be happening
> because there is only one connection. (Granted, it is hard to tell if
> the logging is indicating a threading issue, or if the "Connection
> Reset" error caused the logging abnormality.)
>
> I don't need to block around the "httpClient.execute(request);" do I?
>
> I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> --
>
> I do this to establish the connection:
>
> This is run once in the setup
> -----------------------------
> connectionManager = new
> PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
>
>
>
> This is called when I want to make a request
> --------------------------------------------
> response = httpClient.execute(request);
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 5:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Well, the user went all day without any errors while using the Java.net connector.
>
> The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
>
> Mark
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:47 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Oh, I forgot to add that all my requests are POST requests.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
>
> However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
>
> Mark
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
>
> > It may well be that HttpURLConnection silently retries failed
> > requests
> Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
>
> She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
>
> public boolean isStale() {
> if (!isOpen()) {
> return true;
> }
> if (isEof()) {
> return true;
> }
> try {
> this.inbuffer.isDataAvailable(1);
> return isEof();
> } catch (SocketTimeoutException ex) {
> return false;
> } catch (IOException ex) {
> return true;
> }
> }
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> >
>
> It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
>
>
> > Some other things of note:
> > - She is using JRE 6 Update 13. (There is a long story behind
> > this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),ver
> > if
> > ier);
> >
>
> I do not thing any of these should matter.
>
> Oleg
>
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and
> > > speed does not seem to be a factory (especially with 4.2.3)
> > >
> > > I got in configured and distributed to this user and she is still
> > > having the same problem. Any other ideas? I double checked my
> > > code, and it looks like everything is correct. (She is not using
> > > a
> > > proxy.)
> > >
> >
> > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> >
> > Oleg
> >
> >
> > >
> > > Method that makes the request (from multiple threads)
> > > {
> > > response = httpClient.execute(request);
> > > }
> > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > {
> > > HttpParams params = new BasicHttpParams();
> > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > configureProxy(params,protocol);
> > >
> > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > }
> > > private void configureProxy(HttpParams params, String scheme) {
> > > Proxy proxy = getProxy();
> > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > //SSL or not, we set the proxy up as HTTP
> > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > if (!scheme.equals(SCHEME_HTTP)) {
> > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > }
> > > }
> > >
> > > }
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > >
> > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > >
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHEC
> > > > K,
> > > > Boolean.TRUE);
> > > >
> > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > >
> > > > Mark
> > > >
> > >
> > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > >
> > > Oleg
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > >
> > > > I have a user getting a lot of Connection Reset errors. I did
> > > > not think this had do to with HttpClient, so much as other
> > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > >
> > > > This user is fine for a while, and then will get the Connection
> > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > <Connector
> > > > port="5502"
> > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > server="Unknown"
> > > > connectionLinger="0"
> > > > socket.soTimeout="300000"
> > > > SSLEnabled="true"
> > > > maxThreads="150"
> > > > scheme="https"
> > > > secure="true"
> > > > clientAuth="false"
> > > > keystoreFile="-----"
> > > > keystoreType="PKCS12"
> > > > keystorePass="-----"
> > > > sslProtocol="TLS"
> > > >
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,T
> > > > LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES
> > > > _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > >
> > > >
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > Message (Connection reset)
> > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
I was able to put a break point in DefaultClientConnection.close(). It sometimes gets called when a connection is releasedManagedConnection is called. When the Connection Reset happens, it gets called 3 times in quick succession. However, looks like this is just normal processing for the failed connection. Whatever error is the root cause seems to have already occurred.
The Connection Reset calls do not seem to be correlated to the normal releasedManagedConnection calls that work.
First call to close()
----------------------
DefaultClientConnection.close:168
ManagedClientConnectionImpl.close:127
HttpRequestExecutor.closeConnection:144
HttpRequestExecutor.execute:131
DefaultRequestDirector.tryExecute:717
DefaultRequestDirector.execute:522
AbstractHttpClient.execute:906
AbstractHttpClient.execute:805
AbstractHttpClient.execute:784
TestNetwork$Worker.connect:187
TestNetwork$Worker.run:153
Thread.run:662
Second call to close()
----------------------
DefaultClientConnection.close:168
ManagedClientConnectionImpl.close:127
DefaultRequestDirector.tryExecute:723
DefaultRequestDirector.execute:522
AbstractHttpClient.execute:906
AbstractHttpClient.execute:805
AbstractHttpClient.execute:784
TestNetwork$Worker.connect:187
TestNetwork$Worker.run:153
Thread.run:662
Third call to close()
----------------------
DefaultClientConnection.close:168
HttpPoolEntry.close:89
AbstractConnPool.release:323
PoolingClientConnectionManager.releaseConnection:278
ManagedClientConnectionImpl.abortConnection:463
DefaultRequestDirector.abortConnection:1157
DefaultRequestDirector.execute:621
AbstractHttpClient.execute:906
AbstractHttpClient.execute:805
AbstractHttpClient.execute:784
TestNetwork$Worker.connect:187
TestNetwork$Worker.run:153
Thread.run:662
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:41 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
Below is the last part of what I got.
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection [id: 0][route: {s}->https://x86test.dev.donnell.com:5502] can be kept alive indefinitely Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager closeExpiredConnections
FINE: Closing expired connections
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager requestConnection
FINE: Connection request: [route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager leaseConnection
FINE: Connection leased: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
FINE: Stale connection check
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAddCookies process
FINE: CookieSpec selected: best-match
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process
FINE: Auth cache not set in the context
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestTargetAuthentication process
FINE: Target auth state: UNCHALLENGED
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestProxyAuthentication process
FINE: Proxy auth state: UNCHALLENGED
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
FINE: Attempt 1 to execute request
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: Sending request: POST / HTTP/1.1
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "POST / HTTP/1.1[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Type: binary[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Length: 34[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Connection: Keep-Alive[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> POST / HTTP/1.1
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Type: binary
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Length: 34
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Connection: Keep-Alive
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
FINE: Closing the connection.
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection shutdown
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 0 of 1; total allocated: 0 of 1]
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
at java.lang.Thread.run(Thread.java:662)
Java Result: 1
BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:25 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:07 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Before the upgrade, she was using HttpClient 4.01.
---
Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
The website it is connecting to is just a Tomcat server running the default ROOT webapp. It is delivering the default HTML page we have set up on that. I would imagine that it is ignoring the POST and treating it like a GET
In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 25, 2013 7:46 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
Mark
You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
There are probably several things you might want to try out
(1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
(2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
(3) Otherwise there is no way around getting very close and personal with Wireshark.
Oleg
On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
>
> Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Thursday, January 24, 2013 10:29 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
>
> So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
>
> The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> --
>
> Another avenue I am investigating is trying to force the issue by making the connections work harder.
> I am trying to force the issue doing a lot of requests all at once. I
> reduced my MAX down to 1 to force contention. Everything seems to
> work pretty well. However, I did get the Connection Reset error twice
> on my machine! The logs were fairly regular looking since there is
> only one connection at a time. However, when this happened, it looks
> like it is sending 2 request at once. This shouldn't be happening
> because there is only one connection. (Granted, it is hard to tell if
> the logging is indicating a threading issue, or if the "Connection
> Reset" error caused the logging abnormality.)
>
> I don't need to block around the "httpClient.execute(request);" do I?
>
> I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> --
>
> I do this to establish the connection:
>
> This is run once in the setup
> -----------------------------
> connectionManager = new
> PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
>
>
>
> This is called when I want to make a request
> --------------------------------------------
> response = httpClient.execute(request);
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 5:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Well, the user went all day without any errors while using the Java.net connector.
>
> The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
>
> Mark
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:47 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Oh, I forgot to add that all my requests are POST requests.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
>
> However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
>
> Mark
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
>
> > It may well be that HttpURLConnection silently retries failed
> > requests
> Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
>
> She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
>
> public boolean isStale() {
> if (!isOpen()) {
> return true;
> }
> if (isEof()) {
> return true;
> }
> try {
> this.inbuffer.isDataAvailable(1);
> return isEof();
> } catch (SocketTimeoutException ex) {
> return false;
> } catch (IOException ex) {
> return true;
> }
> }
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> >
>
> It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
>
>
> > Some other things of note:
> > - She is using JRE 6 Update 13. (There is a long story behind
> > this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),ver
> > if
> > ier);
> >
>
> I do not thing any of these should matter.
>
> Oleg
>
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and
> > > speed does not seem to be a factory (especially with 4.2.3)
> > >
> > > I got in configured and distributed to this user and she is still
> > > having the same problem. Any other ideas? I double checked my
> > > code, and it looks like everything is correct. (She is not using
> > > a
> > > proxy.)
> > >
> >
> > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> >
> > Oleg
> >
> >
> > >
> > > Method that makes the request (from multiple threads)
> > > {
> > > response = httpClient.execute(request);
> > > }
> > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > {
> > > HttpParams params = new BasicHttpParams();
> > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > configureProxy(params,protocol);
> > >
> > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > }
> > > private void configureProxy(HttpParams params, String scheme) {
> > > Proxy proxy = getProxy();
> > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > //SSL or not, we set the proxy up as HTTP
> > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > if (!scheme.equals(SCHEME_HTTP)) {
> > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > }
> > > }
> > >
> > > }
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > >
> > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > >
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHEC
> > > > K,
> > > > Boolean.TRUE);
> > > >
> > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > >
> > > > Mark
> > > >
> > >
> > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > >
> > > Oleg
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > >
> > > > I have a user getting a lot of Connection Reset errors. I did
> > > > not think this had do to with HttpClient, so much as other
> > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > >
> > > > This user is fine for a while, and then will get the Connection
> > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > <Connector
> > > > port="5502"
> > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > server="Unknown"
> > > > connectionLinger="0"
> > > > socket.soTimeout="300000"
> > > > SSLEnabled="true"
> > > > maxThreads="150"
> > > > scheme="https"
> > > > secure="true"
> > > > clientAuth="false"
> > > > keystoreFile="-----"
> > > > keystoreType="PKCS12"
> > > > keystorePass="-----"
> > > > sslProtocol="TLS"
> > > >
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,T
> > > > LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES
> > > > _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > >
> > > >
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > Message (Connection reset)
> > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 (pool size). So, I had just 1 thread making the request over and over again without pausing.
Below is the last part of what I got.
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection [id: 0][route: {s}->https://x86test.dev.donnell.com:5502] can be kept alive indefinitely
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1]
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager closeExpiredConnections
FINE: Closing expired connections
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager requestConnection
FINE: Connection request: [route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route allocated: 1 of 1; total allocated: 1 of 1]
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager leaseConnection
FINE: Connection leased: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 1 of 1; total allocated: 1 of 1]
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute
FINE: Stale connection check
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAddCookies process
FINE: CookieSpec selected: best-match
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process
FINE: Auth cache not set in the context
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestTargetAuthentication process
FINE: Target auth state: UNCHALLENGED
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.client.protocol.RequestProxyAuthentication process
FINE: Proxy auth state: UNCHALLENGED
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
FINE: Attempt 1 to execute request
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: Sending request: POST / HTTP/1.1
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "POST / HTTP/1.1[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Type: binary[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Content-Length: 34[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "Connection: Keep-Alive[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "[\r][\n]"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> POST / HTTP/1.1
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Type: binary
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Content-Length: 34
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Host: x86test.dev.donnell.com:5502
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader
FINE: >> Connection: Keep-Alive
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire
FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX"
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector tryExecute
FINE: Closing the connection.
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection shutdown
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection close
FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed
Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager releaseConnection
FINE: Connection released: [id: 0][route: {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route allocated: 0 of 1; total allocated: 0 of 1]
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405)
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187)
at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153)
at java.lang.Thread.run(Thread.java:662)
Java Result: 1
BUILD SUCCESSFUL (total time: 26 seconds)
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:25 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:07 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Before the upgrade, she was using HttpClient 4.01.
---
Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
The website it is connecting to is just a Tomcat server running the default ROOT webapp. It is delivering the default HTML page we have set up on that. I would imagine that it is ignoring the POST and treating it like a GET
In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 25, 2013 7:46 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
Mark
You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
There are probably several things you might want to try out
(1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
(2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
(3) Otherwise there is no way around getting very close and personal with Wireshark.
Oleg
On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
>
> Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Thursday, January 24, 2013 10:29 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
>
> So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
>
> The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> --
>
> Another avenue I am investigating is trying to force the issue by making the connections work harder.
> I am trying to force the issue doing a lot of requests all at once. I
> reduced my MAX down to 1 to force contention. Everything seems to
> work pretty well. However, I did get the Connection Reset error twice
> on my machine! The logs were fairly regular looking since there is
> only one connection at a time. However, when this happened, it looks
> like it is sending 2 request at once. This shouldn't be happening
> because there is only one connection. (Granted, it is hard to tell if
> the logging is indicating a threading issue, or if the "Connection
> Reset" error caused the logging abnormality.)
>
> I don't need to block around the "httpClient.execute(request);" do I?
>
> I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> --
>
> I do this to establish the connection:
>
> This is run once in the setup
> -----------------------------
> connectionManager = new
> PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
>
>
>
> This is called when I want to make a request
> --------------------------------------------
> response = httpClient.execute(request);
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 5:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Well, the user went all day without any errors while using the Java.net connector.
>
> The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
>
> Mark
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:47 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Oh, I forgot to add that all my requests are POST requests.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
>
> However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
>
> Mark
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
>
> > It may well be that HttpURLConnection silently retries failed
> > requests
> Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
>
> She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
>
> public boolean isStale() {
> if (!isOpen()) {
> return true;
> }
> if (isEof()) {
> return true;
> }
> try {
> this.inbuffer.isDataAvailable(1);
> return isEof();
> } catch (SocketTimeoutException ex) {
> return false;
> } catch (IOException ex) {
> return true;
> }
> }
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> >
>
> It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
>
>
> > Some other things of note:
> > - She is using JRE 6 Update 13. (There is a long story behind
> > this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),ver
> > if
> > ier);
> >
>
> I do not thing any of these should matter.
>
> Oleg
>
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and
> > > speed does not seem to be a factory (especially with 4.2.3)
> > >
> > > I got in configured and distributed to this user and she is still
> > > having the same problem. Any other ideas? I double checked my
> > > code, and it looks like everything is correct. (She is not using
> > > a
> > > proxy.)
> > >
> >
> > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> >
> > Oleg
> >
> >
> > >
> > > Method that makes the request (from multiple threads)
> > > {
> > > response = httpClient.execute(request);
> > > }
> > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > {
> > > HttpParams params = new BasicHttpParams();
> > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > configureProxy(params,protocol);
> > >
> > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > }
> > > private void configureProxy(HttpParams params, String scheme) {
> > > Proxy proxy = getProxy();
> > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > //SSL or not, we set the proxy up as HTTP
> > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > if (!scheme.equals(SCHEME_HTTP)) {
> > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > }
> > > }
> > >
> > > }
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > >
> > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > >
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHEC
> > > > K,
> > > > Boolean.TRUE);
> > > >
> > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > >
> > > > Mark
> > > >
> > >
> > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > >
> > > Oleg
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > >
> > > > I have a user getting a lot of Connection Reset errors. I did
> > > > not think this had do to with HttpClient, so much as other
> > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > >
> > > > This user is fine for a while, and then will get the Connection
> > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > <Connector
> > > > port="5502"
> > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > server="Unknown"
> > > > connectionLinger="0"
> > > > socket.soTimeout="300000"
> > > > SSLEnabled="true"
> > > > maxThreads="150"
> > > > scheme="https"
> > > > secure="true"
> > > > clientAuth="false"
> > > > keystoreFile="-----"
> > > > keystoreType="PKCS12"
> > > > keystorePass="-----"
> > > > sslProtocol="TLS"
> > > >
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,T
> > > > LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES
> > > > _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > >
> > > >
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > Message (Connection reset)
> > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
I am running the test using a non-SSL connection and the Connection Reset is not happening. I am going to continue to run the test, but none of the SSL tests ran this long without failing.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 25, 2013 11:07 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Before the upgrade, she was using HttpClient 4.01.
---
Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
The website it is connecting to is just a Tomcat server running the default ROOT webapp. It is delivering the default HTML page we have set up on that. I would imagine that it is ignoring the POST and treating it like a GET
In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 25, 2013 7:46 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
Mark
You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
There are probably several things you might want to try out
(1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
(2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
(3) Otherwise there is no way around getting very close and personal with Wireshark.
Oleg
On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
>
> Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Thursday, January 24, 2013 10:29 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
>
> So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
>
> The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> --
>
> Another avenue I am investigating is trying to force the issue by making the connections work harder.
> I am trying to force the issue doing a lot of requests all at once. I
> reduced my MAX down to 1 to force contention. Everything seems to
> work pretty well. However, I did get the Connection Reset error twice
> on my machine! The logs were fairly regular looking since there is
> only one connection at a time. However, when this happened, it looks
> like it is sending 2 request at once. This shouldn't be happening
> because there is only one connection. (Granted, it is hard to tell if
> the logging is indicating a threading issue, or if the "Connection
> Reset" error caused the logging abnormality.)
>
> I don't need to block around the "httpClient.execute(request);" do I?
>
> I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> --
>
> I do this to establish the connection:
>
> This is run once in the setup
> -----------------------------
> connectionManager = new
> PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
>
>
>
> This is called when I want to make a request
> --------------------------------------------
> response = httpClient.execute(request);
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 5:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Well, the user went all day without any errors while using the Java.net connector.
>
> The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
>
> Mark
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:47 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Oh, I forgot to add that all my requests are POST requests.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
>
> However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
>
> Mark
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
>
> > It may well be that HttpURLConnection silently retries failed
> > requests
> Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
>
> She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
>
> public boolean isStale() {
> if (!isOpen()) {
> return true;
> }
> if (isEof()) {
> return true;
> }
> try {
> this.inbuffer.isDataAvailable(1);
> return isEof();
> } catch (SocketTimeoutException ex) {
> return false;
> } catch (IOException ex) {
> return true;
> }
> }
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> >
>
> It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
>
>
> > Some other things of note:
> > - She is using JRE 6 Update 13. (There is a long story behind
> > this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),ver
> > if
> > ier);
> >
>
> I do not thing any of these should matter.
>
> Oleg
>
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and
> > > speed does not seem to be a factory (especially with 4.2.3)
> > >
> > > I got in configured and distributed to this user and she is still
> > > having the same problem. Any other ideas? I double checked my
> > > code, and it looks like everything is correct. (She is not using
> > > a
> > > proxy.)
> > >
> >
> > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> >
> > Oleg
> >
> >
> > >
> > > Method that makes the request (from multiple threads)
> > > {
> > > response = httpClient.execute(request);
> > > }
> > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > {
> > > HttpParams params = new BasicHttpParams();
> > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > configureProxy(params,protocol);
> > >
> > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > }
> > > private void configureProxy(HttpParams params, String scheme) {
> > > Proxy proxy = getProxy();
> > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > //SSL or not, we set the proxy up as HTTP
> > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > if (!scheme.equals(SCHEME_HTTP)) {
> > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > }
> > > }
> > >
> > > }
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > >
> > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > >
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHEC
> > > > K,
> > > > Boolean.TRUE);
> > > >
> > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > >
> > > > Mark
> > > >
> > >
> > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > >
> > > Oleg
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > >
> > > > I have a user getting a lot of Connection Reset errors. I did
> > > > not think this had do to with HttpClient, so much as other
> > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > >
> > > > This user is fine for a while, and then will get the Connection
> > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > <Connector
> > > > port="5502"
> > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > server="Unknown"
> > > > connectionLinger="0"
> > > > socket.soTimeout="300000"
> > > > SSLEnabled="true"
> > > > maxThreads="150"
> > > > scheme="https"
> > > > secure="true"
> > > > clientAuth="false"
> > > > keystoreFile="-----"
> > > > keystoreType="PKCS12"
> > > > keystorePass="-----"
> > > > sslProtocol="TLS"
> > > >
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,T
> > > > LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES
> > > > _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > >
> > > >
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > Message (Connection reset)
> > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Before the upgrade, she was using HttpClient 4.01.
---
Not sure if it is kosher to send an attachment to the list, but here is a test program. This errors for me after just a minute or two.
The website it is connecting to is just a Tomcat server running the default ROOT webapp. It is delivering the default HTML page we have set up on that. I would imagine that it is ignoring the POST and treating it like a GET
In production, where she is having the problem, it is using the default HttpsURLConnection socket factory. However, for testing, I am using this one to bypass the normal certificate checks and stuff. It happens to both of us, so I don't think the error has anything to do with the socket factory. The message that we send in production are also a lot longer, so I don't think it has anything to do with message size either.
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 25, 2013 7:46 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
Mark
You do not need to do anything special other than closing the content input stream of the response entity (if it is present) to ensure proper resource deallocation. What is important that it has to be done no matter if you need the content or not. I do not think the problem you are dealing with has anything to do with resource management.
There are probably several things you might want to try out
(1) if you can confirm that this issue is HttpClient version specific and tell me what version of HttpClient exhibited it first (say, after upgrade from 4.0.3 to 4.1) I could analyze the differences and try to figure out what could potentially be the cause of connection resets.
(2) This is entirely possible this is somehow SSL related and connection gets reset by the SSL stack. You might want to run your app with SSL debug enabled and see if there is anything in the SSL logs that could help explain connection being reset.
(3) Otherwise there is no way around getting very close and personal with Wireshark.
Oleg
On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
>
> Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Thursday, January 24, 2013 10:29 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
>
> So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
>
> The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> --
>
> Another avenue I am investigating is trying to force the issue by making the connections work harder.
> I am trying to force the issue doing a lot of requests all at once. I
> reduced my MAX down to 1 to force contention. Everything seems to
> work pretty well. However, I did get the Connection Reset error twice
> on my machine! The logs were fairly regular looking since there is
> only one connection at a time. However, when this happened, it looks
> like it is sending 2 request at once. This shouldn't be happening
> because there is only one connection. (Granted, it is hard to tell if
> the logging is indicating a threading issue, or if the "Connection
> Reset" error caused the logging abnormality.)
>
> I don't need to block around the "httpClient.execute(request);" do I?
>
> I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> --
>
> I do this to establish the connection:
>
> This is run once in the setup
> -----------------------------
> connectionManager = new
> PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
>
>
>
> This is called when I want to make a request
> --------------------------------------------
> response = httpClient.execute(request);
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 5:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Well, the user went all day without any errors while using the Java.net connector.
>
> The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
>
> Mark
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:47 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Oh, I forgot to add that all my requests are POST requests.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
>
> However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
>
> Mark
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
>
> > It may well be that HttpURLConnection silently retries failed
> > requests
> Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
>
> She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
>
> public boolean isStale() {
> if (!isOpen()) {
> return true;
> }
> if (isEof()) {
> return true;
> }
> try {
> this.inbuffer.isDataAvailable(1);
> return isEof();
> } catch (SocketTimeoutException ex) {
> return false;
> } catch (IOException ex) {
> return true;
> }
> }
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> >
>
> It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
>
>
> > Some other things of note:
> > - She is using JRE 6 Update 13. (There is a long story behind
> > this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),ver
> > if
> > ier);
> >
>
> I do not thing any of these should matter.
>
> Oleg
>
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and
> > > speed does not seem to be a factory (especially with 4.2.3)
> > >
> > > I got in configured and distributed to this user and she is still
> > > having the same problem. Any other ideas? I double checked my
> > > code, and it looks like everything is correct. (She is not using
> > > a
> > > proxy.)
> > >
> >
> > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> >
> > Oleg
> >
> >
> > >
> > > Method that makes the request (from multiple threads)
> > > {
> > > response = httpClient.execute(request);
> > > }
> > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > {
> > > HttpParams params = new BasicHttpParams();
> > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > configureProxy(params,protocol);
> > >
> > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > }
> > > private void configureProxy(HttpParams params, String scheme) {
> > > Proxy proxy = getProxy();
> > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > //SSL or not, we set the proxy up as HTTP
> > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > if (!scheme.equals(SCHEME_HTTP)) {
> > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > }
> > > }
> > >
> > > }
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > >
> > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > >
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHEC
> > > > K,
> > > > Boolean.TRUE);
> > > >
> > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > >
> > > > Mark
> > > >
> > >
> > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > >
> > > Oleg
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > >
> > > > I have a user getting a lot of Connection Reset errors. I did
> > > > not think this had do to with HttpClient, so much as other
> > > > factors on her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > >
> > > > This user is fine for a while, and then will get the Connection
> > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > <Connector
> > > > port="5502"
> > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > server="Unknown"
> > > > connectionLinger="0"
> > > > socket.soTimeout="300000"
> > > > SSLEnabled="true"
> > > > maxThreads="150"
> > > > scheme="https"
> > > > secure="true"
> > > > clientAuth="false"
> > > > keystoreFile="-----"
> > > > keystoreType="PKCS12"
> > > > keystorePass="-----"
> > > > sslProtocol="TLS"
> > > >
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,T
> > > > LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES
> > > > _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > >
> > > >
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > Message (Connection reset)
> > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > >
> > > >
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ----------------------------------------------------------------
> > > > --
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
Mark
You do not need to do anything special other than closing the content
input stream of the response entity (if it is present) to ensure proper
resource deallocation. What is important that it has to be done no
matter if you need the content or not. I do not think the problem you
are dealing with has anything to do with resource management.
There are probably several things you might want to try out
(1) if you can confirm that this issue is HttpClient version specific
and tell me what version of HttpClient exhibited it first (say, after
upgrade from 4.0.3 to 4.1) I could analyze the differences and try to
figure out what could potentially be the cause of connection resets.
(2) This is entirely possible this is somehow SSL related and connection
gets reset by the SSL stack. You might want to run your app with SSL
debug enabled and see if there is anything in the SSL logs that could
help explain connection being reset.
(3) Otherwise there is no way around getting very close and personal
with Wireshark.
Oleg
On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote:
> Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
>
> Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Thursday, January 24, 2013 10:29 AM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
>
> So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
>
> The docs say the closing the InputStream will do it, but will anything else do it implicitly?
> --
>
> Another avenue I am investigating is trying to force the issue by making the connections work harder.
> I am trying to force the issue doing a lot of requests all at once. I reduced my MAX down to 1 to force contention. Everything seems to work pretty well. However, I did get the Connection Reset error twice on my machine! The logs were fairly regular looking since there is only one connection at a time. However, when this happened, it looks like it is sending 2 request at once. This shouldn't be happening because there is only one connection. (Granted, it is hard to tell if the logging is indicating a threading issue, or if the "Connection Reset" error caused the logging abnormality.)
>
> I don't need to block around the "httpClient.execute(request);" do I?
>
> I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
> --
>
> I do this to establish the connection:
>
> This is run once in the setup
> -----------------------------
> connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
>
>
>
> This is called when I want to make a request
> --------------------------------------------
> response = httpClient.execute(request);
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 5:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Well, the user went all day without any errors while using the Java.net connector.
>
> The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
>
> Mark
>
>
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:47 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Oh, I forgot to add that all my requests are POST requests.
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 1:02 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
>
> However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
>
> Mark
>
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Friday, January 18, 2013 12:06 PM
> To: 'HttpClient User Discussion'
> Subject: RE: Connection Reset errors
>
> Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
>
> > It may well be that HttpURLConnection silently retries failed requests
> Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
>
> She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
>
> public boolean isStale() {
> if (!isOpen()) {
> return true;
> }
> if (isEof()) {
> return true;
> }
> try {
> this.inbuffer.isDataAvailable(1);
> return isEof();
> } catch (SocketTimeoutException ex) {
> return false;
> } catch (IOException ex) {
> return true;
> }
> }
>
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 11:24 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> > Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
> >
>
> It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
>
>
> > Some other things of note:
> > - She is using JRE 6 Update 13. (There is a long story behind this.)
> > - The application is launched via Webstart
> > - The SocketFactory is created with: fact = new
> > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> > ier);
> >
>
> I do not thing any of these should matter.
>
> Oleg
>
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Friday, January 18, 2013 6:17 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > > Thanks for the explanation.
> > > > pro-actively evicting expired connections and connections that
> > > > have been idle
> > > Seems like just doing the stale check would more reliable, and speed
> > > does not seem to be a factory (especially with 4.2.3)
> > >
> > > I got in configured and distributed to this user and she is still
> > > having the same problem. Any other ideas? I double checked my
> > > code, and it looks like everything is correct. (She is not using a
> > > proxy.)
> > >
> >
> > This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
> >
> > Oleg
> >
> >
> > >
> > > Method that makes the request (from multiple threads)
> > > {
> > > response = httpClient.execute(request);
> > > }
> > > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > > {
> > > HttpParams params = new BasicHttpParams();
> > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > > configureProxy(params,protocol);
> > >
> > > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > > connectionManager.setDefaultMaxPerRoute(MAX);
> > > connectionManager.setMaxTotal(MAX);
> > >
> > > httpClient = new DefaultHttpClient(connectionManager, params);
> > > }
> > > private void configureProxy(HttpParams params, String scheme) {
> > > Proxy proxy = getProxy();
> > > if (proxy.type() != Proxy.Type.DIRECT) {
> > > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > > //SSL or not, we set the proxy up as HTTP
> > > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > > if (!scheme.equals(SCHEME_HTTP)) {
> > > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > > }
> > > }
> > >
> > > }
> > >
> > > -----Original Message-----
> > > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > > Sent: Thursday, January 17, 2013 7:21 AM
> > > To: HttpClient User Discussion
> > > Subject: Re: Connection Reset errors
> > >
> > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> > >
> > > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> > >
> > > > I noticed that there is a stale connection check:
> > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > > Boolean.TRUE);
> > > >
> > > > Is this something that I should be using? I am connecting to the same source over and over again.
> > > >
> > > > Mark
> > > >
> > >
> > > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> > >
> > > Oleg
> > >
> > >
> > > > -----Original Message-----
> > > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > > To: 'HttpClient User Discussion'
> > > > Subject: Connection Reset errors
> > > >
> > > > I have a user getting a lot of Connection Reset errors. I did not
> > > > think this had do to with HttpClient, so much as other factors on
> > > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > > >
> > > > This user is fine for a while, and then will get the Connection
> > > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Mark
> > > >
> > > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > > <Connector
> > > > port="5502"
> > > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > > server="Unknown"
> > > > connectionLinger="0"
> > > > socket.soTimeout="300000"
> > > > SSLEnabled="true"
> > > > maxThreads="150"
> > > > scheme="https"
> > > > secure="true"
> > > > clientAuth="false"
> > > > keystoreFile="-----"
> > > > keystoreType="PKCS12"
> > > > keystorePass="-----"
> > > > sslProtocol="TLS"
> > > >
> > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > > >
> > > >
> > > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > > Message (Connection reset)
> > > > at java.net.SocketInputStream.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > > at
> > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > > ja
> > > > va:138)
> > > >
> > > >
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > > >
> > > > ------------------------------------------------------------------
> > > > --
> > > > - To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > >
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Something else I thought of is this: We have some transactions where the response is not too important. For instance, if we send a pulse to the server to let it know we are still here. We don't really care what the response is, if we didn't get an error then we did our part. If we did get and error, we will try again soon anyway, so no big deal.
Is there something that needs to be done on a connection that is closed before reading all the data? I have never experienced reading stale data, so this seems a bit unlikely to me. However, if there is implicit cleanup that is being done, could this be, in some way, causing my problems?
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Thursday, January 24, 2013 10:29 AM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
The docs say the closing the InputStream will do it, but will anything else do it implicitly?
--
Another avenue I am investigating is trying to force the issue by making the connections work harder.
I am trying to force the issue doing a lot of requests all at once. I reduced my MAX down to 1 to force contention. Everything seems to work pretty well. However, I did get the Connection Reset error twice on my machine! The logs were fairly regular looking since there is only one connection at a time. However, when this happened, it looks like it is sending 2 request at once. This shouldn't be happening because there is only one connection. (Granted, it is hard to tell if the logging is indicating a threading issue, or if the "Connection Reset" error caused the logging abnormality.)
I don't need to block around the "httpClient.execute(request);" do I?
I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
--
I do this to establish the connection:
This is run once in the setup
-----------------------------
connectionManager = new PoolingClientConnectionManager(schemeRegistry);
connectionManager.setDefaultMaxPerRoute(MAX);
connectionManager.setMaxTotal(MAX);
httpClient = new DefaultHttpClient(connectionManager, params);
This is called when I want to make a request
--------------------------------------------
response = httpClient.execute(request);
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 5:06 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Well, the user went all day without any errors while using the Java.net connector.
The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 1:47 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Oh, I forgot to add that all my requests are POST requests.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 1:02 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 12:06 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> It may well be that HttpURLConnection silently retries failed requests
Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
public boolean isStale() {
if (!isOpen()) {
return true;
}
if (isEof()) {
return true;
}
try {
this.inbuffer.isDataAvailable(1);
return isEof();
} catch (SocketTimeoutException ex) {
return false;
} catch (IOException ex) {
return true;
}
}
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 18, 2013 11:24 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
>
It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> Some other things of note:
> - She is using JRE 6 Update 13. (There is a long story behind this.)
> - The application is launched via Webstart
> - The SocketFactory is created with: fact = new
> SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> ier);
>
I do not thing any of these should matter.
Oleg
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 6:17 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > Thanks for the explanation.
> > > pro-actively evicting expired connections and connections that
> > > have been idle
> > Seems like just doing the stale check would more reliable, and speed
> > does not seem to be a factory (especially with 4.2.3)
> >
> > I got in configured and distributed to this user and she is still
> > having the same problem. Any other ideas? I double checked my
> > code, and it looks like everything is correct. (She is not using a
> > proxy.)
> >
>
> This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
>
> Oleg
>
>
> >
> > Method that makes the request (from multiple threads)
> > {
> > response = httpClient.execute(request);
> > }
> > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > {
> > HttpParams params = new BasicHttpParams();
> > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > configureProxy(params,protocol);
> >
> > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> > }
> > private void configureProxy(HttpParams params, String scheme) {
> > Proxy proxy = getProxy();
> > if (proxy.type() != Proxy.Type.DIRECT) {
> > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > //SSL or not, we set the proxy up as HTTP
> > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > if (!scheme.equals(SCHEME_HTTP)) {
> > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > }
> > }
> >
> > }
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, January 17, 2013 7:21 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> >
> > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> >
> > > I noticed that there is a stale connection check:
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > Boolean.TRUE);
> > >
> > > Is this something that I should be using? I am connecting to the same source over and over again.
> > >
> > > Mark
> > >
> >
> > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> >
> > Oleg
> >
> >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: Connection Reset errors
> > >
> > > I have a user getting a lot of Connection Reset errors. I did not
> > > think this had do to with HttpClient, so much as other factors on
> > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > >
> > > This user is fine for a while, and then will get the Connection
> > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > <Connector
> > > port="5502"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > server="Unknown"
> > > connectionLinger="0"
> > > socket.soTimeout="300000"
> > > SSLEnabled="true"
> > > maxThreads="150"
> > > scheme="https"
> > > secure="true"
> > > clientAuth="false"
> > > keystoreFile="-----"
> > > keystoreType="PKCS12"
> > > keystorePass="-----"
> > > sslProtocol="TLS"
> > >
> > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > >
> > >
> > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > Message (Connection reset)
> > > at java.net.SocketInputStream.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > ja
> > > va:138)
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
I am just not convinced this is a network thing. It happens too often for her, and never with the java.net connection. She has a newer machine, maybe threading issues are just more apparent on her hardware.
So, maybe the problem is somewhere in my code. Can you tell me what are the methods that might release a connection back to the pool? I am wondering if there is a way I could be releasing the connection back to the pool, it gets picked up by something else, and then the original thread tries to close it again or something.
The docs say the closing the InputStream will do it, but will anything else do it implicitly?
--
Another avenue I am investigating is trying to force the issue by making the connections work harder.
I am trying to force the issue doing a lot of requests all at once. I reduced my MAX down to 1 to force contention. Everything seems to work pretty well. However, I did get the Connection Reset error twice on my machine! The logs were fairly regular looking since there is only one connection at a time. However, when this happened, it looks like it is sending 2 request at once. This shouldn't be happening because there is only one connection. (Granted, it is hard to tell if the logging is indicating a threading issue, or if the "Connection Reset" error caused the logging abnormality.)
I don't need to block around the "httpClient.execute(request);" do I?
I am using HttpClient 4.2.3 (with HttpCore 4.2.3)
--
I do this to establish the connection:
This is run once in the setup
-----------------------------
connectionManager = new PoolingClientConnectionManager(schemeRegistry);
connectionManager.setDefaultMaxPerRoute(MAX);
connectionManager.setMaxTotal(MAX);
httpClient = new DefaultHttpClient(connectionManager, params);
This is called when I want to make a request
--------------------------------------------
response = httpClient.execute(request);
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 5:06 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Well, the user went all day without any errors while using the Java.net connector.
The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 1:47 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Oh, I forgot to add that all my requests are POST requests.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 1:02 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 12:06 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> It may well be that HttpURLConnection silently retries failed requests
Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
public boolean isStale() {
if (!isOpen()) {
return true;
}
if (isEof()) {
return true;
}
try {
this.inbuffer.isDataAvailable(1);
return isEof();
} catch (SocketTimeoutException ex) {
return false;
} catch (IOException ex) {
return true;
}
}
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 18, 2013 11:24 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
>
It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> Some other things of note:
> - She is using JRE 6 Update 13. (There is a long story behind this.)
> - The application is launched via Webstart
> - The SocketFactory is created with: fact = new
> SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> ier);
>
I do not thing any of these should matter.
Oleg
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 6:17 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > Thanks for the explanation.
> > > pro-actively evicting expired connections and connections that
> > > have been idle
> > Seems like just doing the stale check would more reliable, and speed
> > does not seem to be a factory (especially with 4.2.3)
> >
> > I got in configured and distributed to this user and she is still
> > having the same problem. Any other ideas? I double checked my
> > code, and it looks like everything is correct. (She is not using a
> > proxy.)
> >
>
> This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
>
> Oleg
>
>
> >
> > Method that makes the request (from multiple threads)
> > {
> > response = httpClient.execute(request);
> > }
> > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > {
> > HttpParams params = new BasicHttpParams();
> > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > configureProxy(params,protocol);
> >
> > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> > }
> > private void configureProxy(HttpParams params, String scheme) {
> > Proxy proxy = getProxy();
> > if (proxy.type() != Proxy.Type.DIRECT) {
> > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > //SSL or not, we set the proxy up as HTTP
> > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > if (!scheme.equals(SCHEME_HTTP)) {
> > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > }
> > }
> >
> > }
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, January 17, 2013 7:21 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> >
> > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> >
> > > I noticed that there is a stale connection check:
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > Boolean.TRUE);
> > >
> > > Is this something that I should be using? I am connecting to the same source over and over again.
> > >
> > > Mark
> > >
> >
> > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> >
> > Oleg
> >
> >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: Connection Reset errors
> > >
> > > I have a user getting a lot of Connection Reset errors. I did not
> > > think this had do to with HttpClient, so much as other factors on
> > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > >
> > > This user is fine for a while, and then will get the Connection
> > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > <Connector
> > > port="5502"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > server="Unknown"
> > > connectionLinger="0"
> > > socket.soTimeout="300000"
> > > SSLEnabled="true"
> > > maxThreads="150"
> > > scheme="https"
> > > secure="true"
> > > clientAuth="false"
> > > keystoreFile="-----"
> > > keystoreType="PKCS12"
> > > keystorePass="-----"
> > > sslProtocol="TLS"
> > >
> > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > >
> > >
> > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > Message (Connection reset)
> > > at java.net.SocketInputStream.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > ja
> > > va:138)
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Well, the user went all day without any errors while using the Java.net connector.
The error using the HttpClient connector has been happening with HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 (and now is continuing under 4.2.3). I will have to check on the upgrade path and see if she had been using a 4.0.x version before this, or if they jumped from a version that used a 3.x version.
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 1:47 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Oh, I forgot to add that all my requests are POST requests.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 1:02 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 12:06 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> It may well be that HttpURLConnection silently retries failed requests
Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
public boolean isStale() {
if (!isOpen()) {
return true;
}
if (isEof()) {
return true;
}
try {
this.inbuffer.isDataAvailable(1);
return isEof();
} catch (SocketTimeoutException ex) {
return false;
} catch (IOException ex) {
return true;
}
}
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 18, 2013 11:24 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
>
It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> Some other things of note:
> - She is using JRE 6 Update 13. (There is a long story behind this.)
> - The application is launched via Webstart
> - The SocketFactory is created with: fact = new
> SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> ier);
>
I do not thing any of these should matter.
Oleg
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 6:17 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > Thanks for the explanation.
> > > pro-actively evicting expired connections and connections that
> > > have been idle
> > Seems like just doing the stale check would more reliable, and speed
> > does not seem to be a factory (especially with 4.2.3)
> >
> > I got in configured and distributed to this user and she is still
> > having the same problem. Any other ideas? I double checked my
> > code, and it looks like everything is correct. (She is not using a
> > proxy.)
> >
>
> This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
>
> Oleg
>
>
> >
> > Method that makes the request (from multiple threads)
> > {
> > response = httpClient.execute(request);
> > }
> > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > {
> > HttpParams params = new BasicHttpParams();
> > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > configureProxy(params,protocol);
> >
> > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> > }
> > private void configureProxy(HttpParams params, String scheme) {
> > Proxy proxy = getProxy();
> > if (proxy.type() != Proxy.Type.DIRECT) {
> > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > //SSL or not, we set the proxy up as HTTP
> > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > if (!scheme.equals(SCHEME_HTTP)) {
> > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > }
> > }
> >
> > }
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, January 17, 2013 7:21 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> >
> > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> >
> > > I noticed that there is a stale connection check:
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > Boolean.TRUE);
> > >
> > > Is this something that I should be using? I am connecting to the same source over and over again.
> > >
> > > Mark
> > >
> >
> > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> >
> > Oleg
> >
> >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: Connection Reset errors
> > >
> > > I have a user getting a lot of Connection Reset errors. I did not
> > > think this had do to with HttpClient, so much as other factors on
> > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > >
> > > This user is fine for a while, and then will get the Connection
> > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > <Connector
> > > port="5502"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > server="Unknown"
> > > connectionLinger="0"
> > > socket.soTimeout="300000"
> > > SSLEnabled="true"
> > > maxThreads="150"
> > > scheme="https"
> > > secure="true"
> > > clientAuth="false"
> > > keystoreFile="-----"
> > > keystoreType="PKCS12"
> > > keystorePass="-----"
> > > sslProtocol="TLS"
> > >
> > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > >
> > >
> > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > Message (Connection reset)
> > > at java.net.SocketInputStream.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > ja
> > > va:138)
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Oh, I forgot to add that all my requests are POST requests.
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 1:02 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 12:06 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> It may well be that HttpURLConnection silently retries failed requests
Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
public boolean isStale() {
if (!isOpen()) {
return true;
}
if (isEof()) {
return true;
}
try {
this.inbuffer.isDataAvailable(1);
return isEof();
} catch (SocketTimeoutException ex) {
return false;
} catch (IOException ex) {
return true;
}
}
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 18, 2013 11:24 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
>
It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> Some other things of note:
> - She is using JRE 6 Update 13. (There is a long story behind this.)
> - The application is launched via Webstart
> - The SocketFactory is created with: fact = new
> SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> ier);
>
I do not thing any of these should matter.
Oleg
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 6:17 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > Thanks for the explanation.
> > > pro-actively evicting expired connections and connections that
> > > have been idle
> > Seems like just doing the stale check would more reliable, and speed
> > does not seem to be a factory (especially with 4.2.3)
> >
> > I got in configured and distributed to this user and she is still
> > having the same problem. Any other ideas? I double checked my
> > code, and it looks like everything is correct. (She is not using a
> > proxy.)
> >
>
> This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
>
> Oleg
>
>
> >
> > Method that makes the request (from multiple threads)
> > {
> > response = httpClient.execute(request);
> > }
> > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > {
> > HttpParams params = new BasicHttpParams();
> > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > configureProxy(params,protocol);
> >
> > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> > }
> > private void configureProxy(HttpParams params, String scheme) {
> > Proxy proxy = getProxy();
> > if (proxy.type() != Proxy.Type.DIRECT) {
> > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > //SSL or not, we set the proxy up as HTTP
> > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > if (!scheme.equals(SCHEME_HTTP)) {
> > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > }
> > }
> >
> > }
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, January 17, 2013 7:21 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> >
> > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> >
> > > I noticed that there is a stale connection check:
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > Boolean.TRUE);
> > >
> > > Is this something that I should be using? I am connecting to the same source over and over again.
> > >
> > > Mark
> > >
> >
> > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> >
> > Oleg
> >
> >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: Connection Reset errors
> > >
> > > I have a user getting a lot of Connection Reset errors. I did not
> > > think this had do to with HttpClient, so much as other factors on
> > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > >
> > > This user is fine for a while, and then will get the Connection
> > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > <Connector
> > > port="5502"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > server="Unknown"
> > > connectionLinger="0"
> > > socket.soTimeout="300000"
> > > SSLEnabled="true"
> > > maxThreads="150"
> > > scheme="https"
> > > secure="true"
> > > clientAuth="false"
> > > keystoreFile="-----"
> > > keystoreType="PKCS12"
> > > keystorePass="-----"
> > > sslProtocol="TLS"
> > >
> > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > >
> > >
> > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > Message (Connection reset)
> > > at java.net.SocketInputStream.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > ja
> > > va:138)
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
While she is testing the java.net stuff, I decided to look into the possibility of retrying the request. Looking at this more closely, I see that the error is on the reading of the data, and the write seems to succeed. This gives me something to check on, since I should be able to see whether or not the servlet actually got her request or not.
However, is this a common manifestation of the Connection Reset error? The write succeeds (or fails silently) and then it is on the read the problem can be detected? Or does this mean that the socket was fine during the write operation, but then was somehow severed before the read?
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Friday, January 18, 2013 12:06 PM
To: 'HttpClient User Discussion'
Subject: RE: Connection Reset errors
Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> It may well be that HttpURLConnection silently retries failed requests
Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
public boolean isStale() {
if (!isOpen()) {
return true;
}
if (isEof()) {
return true;
}
try {
this.inbuffer.isDataAvailable(1);
return isEof();
} catch (SocketTimeoutException ex) {
return false;
} catch (IOException ex) {
return true;
}
}
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 18, 2013 11:24 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
>
It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> Some other things of note:
> - She is using JRE 6 Update 13. (There is a long story behind this.)
> - The application is launched via Webstart
> - The SocketFactory is created with: fact = new
> SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> ier);
>
I do not thing any of these should matter.
Oleg
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 6:17 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > Thanks for the explanation.
> > > pro-actively evicting expired connections and connections that
> > > have been idle
> > Seems like just doing the stale check would more reliable, and speed
> > does not seem to be a factory (especially with 4.2.3)
> >
> > I got in configured and distributed to this user and she is still
> > having the same problem. Any other ideas? I double checked my
> > code, and it looks like everything is correct. (She is not using a
> > proxy.)
> >
>
> This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
>
> Oleg
>
>
> >
> > Method that makes the request (from multiple threads)
> > {
> > response = httpClient.execute(request);
> > }
> > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > {
> > HttpParams params = new BasicHttpParams();
> > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > configureProxy(params,protocol);
> >
> > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> > }
> > private void configureProxy(HttpParams params, String scheme) {
> > Proxy proxy = getProxy();
> > if (proxy.type() != Proxy.Type.DIRECT) {
> > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > //SSL or not, we set the proxy up as HTTP
> > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > if (!scheme.equals(SCHEME_HTTP)) {
> > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > }
> > }
> >
> > }
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, January 17, 2013 7:21 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> >
> > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> >
> > > I noticed that there is a stale connection check:
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > Boolean.TRUE);
> > >
> > > Is this something that I should be using? I am connecting to the same source over and over again.
> > >
> > > Mark
> > >
> >
> > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> >
> > Oleg
> >
> >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: Connection Reset errors
> > >
> > > I have a user getting a lot of Connection Reset errors. I did not
> > > think this had do to with HttpClient, so much as other factors on
> > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > >
> > > This user is fine for a while, and then will get the Connection
> > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > <Connector
> > > port="5502"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > server="Unknown"
> > > connectionLinger="0"
> > > socket.soTimeout="300000"
> > > SSLEnabled="true"
> > > maxThreads="150"
> > > scheme="https"
> > > secure="true"
> > > clientAuth="false"
> > > keystoreFile="-----"
> > > keystoreType="PKCS12"
> > > keystorePass="-----"
> > > sslProtocol="TLS"
> > >
> > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > >
> > >
> > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > Message (Connection reset)
> > > at java.net.SocketInputStream.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > ja
> > > va:138)
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Looks like the stale connection check is enabled by default anyway, so I must have been using it all along.
> It may well be that HttpURLConnection silently retries failed requests
Is this a common thing to do, like for browsers? Granted, she is in our app more than most others, but I don't think she experiences issues with other applications.
She is not getting a Connection Refused or some other unable to read exception, but a Connection Reset. The stale connection check passed, so HttpClient thinks this is a good connection, right? Or are there limits to how reliable a stale connection check can be?
public boolean isStale() {
if (!isOpen()) {
return true;
}
if (isEof()) {
return true;
}
try {
this.inbuffer.isDataAvailable(1);
return isEof();
} catch (SocketTimeoutException ex) {
return false;
} catch (IOException ex) {
return true;
}
}
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 18, 2013 11:24 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
>
It may well be that HttpURLConnection silently retries failed requests without your knowing about it. It certainly did that when I last touched it (which admittedly was a looooooong time ago), and that was precisely the reason I had to stop using it and to look for a better alternative.
> Some other things of note:
> - She is using JRE 6 Update 13. (There is a long story behind this.)
> - The application is launched via Webstart
> - The SocketFactory is created with: fact = new
> SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verif
> ier);
>
I do not thing any of these should matter.
Oleg
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 6:17 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > Thanks for the explanation.
> > > pro-actively evicting expired connections and connections that
> > > have been idle
> > Seems like just doing the stale check would more reliable, and speed
> > does not seem to be a factory (especially with 4.2.3)
> >
> > I got in configured and distributed to this user and she is still
> > having the same problem. Any other ideas? I double checked my
> > code, and it looks like everything is correct. (She is not using a
> > proxy.)
> >
>
> This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
>
> Oleg
>
>
> >
> > Method that makes the request (from multiple threads)
> > {
> > response = httpClient.execute(request);
> > }
> > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > {
> > HttpParams params = new BasicHttpParams();
> > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > configureProxy(params,protocol);
> >
> > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> > }
> > private void configureProxy(HttpParams params, String scheme) {
> > Proxy proxy = getProxy();
> > if (proxy.type() != Proxy.Type.DIRECT) {
> > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > //SSL or not, we set the proxy up as HTTP
> > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > if (!scheme.equals(SCHEME_HTTP)) {
> > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > }
> > }
> >
> > }
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, January 17, 2013 7:21 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> >
> > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> >
> > > I noticed that there is a stale connection check:
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > Boolean.TRUE);
> > >
> > > Is this something that I should be using? I am connecting to the same source over and over again.
> > >
> > > Mark
> > >
> >
> > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> >
> > Oleg
> >
> >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: Connection Reset errors
> > >
> > > I have a user getting a lot of Connection Reset errors. I did not
> > > think this had do to with HttpClient, so much as other factors on
> > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > >
> > > This user is fine for a while, and then will get the Connection
> > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > <Connector
> > > port="5502"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > server="Unknown"
> > > connectionLinger="0"
> > > socket.soTimeout="300000"
> > > SSLEnabled="true"
> > > maxThreads="150"
> > > scheme="https"
> > > secure="true"
> > > clientAuth="false"
> > > keystoreFile="-----"
> > > keystoreType="PKCS12"
> > > keystorePass="-----"
> > > sslProtocol="TLS"
> > >
> > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS
> > > _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_E
> > > DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > >
> > >
> > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > Message (Connection reset)
> > > at java.net.SocketInputStream.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > ja
> > > va:138)
> > >
> > >
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> > > ------------------------------------------------------------------
> > > --
> > > - To unsubscribe, e-mail:
> > > httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote:
> Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
>
It may well be that HttpURLConnection silently retries failed requests
without your knowing about it. It certainly did that when I last touched
it (which admittedly was a looooooong time ago), and that was precisely
the reason I had to stop using it and to look for a better alternative.
> Some other things of note:
> - She is using JRE 6 Update 13. (There is a long story behind this.)
> - The application is launched via Webstart
> - The SocketFactory is created with: fact = new SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verifier);
>
I do not thing any of these should matter.
Oleg
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Friday, January 18, 2013 6:17 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> > Thanks for the explanation.
> > > pro-actively evicting expired connections and connections that have
> > > been idle
> > Seems like just doing the stale check would more reliable, and speed
> > does not seem to be a factory (especially with 4.2.3)
> >
> > I got in configured and distributed to this user and she is still
> > having the same problem. Any other ideas? I double checked my code,
> > and it looks like everything is correct. (She is not using a proxy.)
> >
>
> This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
>
> Oleg
>
>
> >
> > Method that makes the request (from multiple threads)
> > {
> > response = httpClient.execute(request);
> > }
> > Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> > {
> > HttpParams params = new BasicHttpParams();
> > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> > configureProxy(params,protocol);
> >
> > connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> > connectionManager.setDefaultMaxPerRoute(MAX);
> > connectionManager.setMaxTotal(MAX);
> >
> > httpClient = new DefaultHttpClient(connectionManager, params);
> > }
> > private void configureProxy(HttpParams params, String scheme) {
> > Proxy proxy = getProxy();
> > if (proxy.type() != Proxy.Type.DIRECT) {
> > InetSocketAddress address = (InetSocketAddress) proxy.address();
> > //SSL or not, we set the proxy up as HTTP
> > HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> > LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> > if (!scheme.equals(SCHEME_HTTP)) {
> > //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> > schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> > }
> > }
> >
> > }
> >
> > -----Original Message-----
> > From: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Sent: Thursday, January 17, 2013 7:21 AM
> > To: HttpClient User Discussion
> > Subject: Re: Connection Reset errors
> >
> > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
> >
> > Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> >
> > > I noticed that there is a stale connection check:
> > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > > Boolean.TRUE);
> > >
> > > Is this something that I should be using? I am connecting to the same source over and over again.
> > >
> > > Mark
> > >
> >
> > Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
> >
> > Oleg
> >
> >
> > > -----Original Message-----
> > > From: Mark Claassen [mailto:mac01@donnell.com]
> > > Sent: Wednesday, January 16, 2013 3:07 PM
> > > To: 'HttpClient User Discussion'
> > > Subject: Connection Reset errors
> > >
> > > I have a user getting a lot of Connection Reset errors. I did not
> > > think this had do to with HttpClient, so much as other factors on
> > > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> > >
> > > This user is fine for a while, and then will get the Connection
> > > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > > <Connector
> > > port="5502"
> > > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > > server="Unknown"
> > > connectionLinger="0"
> > > socket.soTimeout="300000"
> > > SSLEnabled="true"
> > > maxThreads="150"
> > > scheme="https"
> > > secure="true"
> > > clientAuth="false"
> > > keystoreFile="-----"
> > > keystoreType="PKCS12"
> > > keystorePass="-----"
> > > sslProtocol="TLS"
> > >
> > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_D
> > > HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE
> > > _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> > >
> > >
> > > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > > Message (Connection reset)
> > > at java.net.SocketInputStream.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > > at
> > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > > ja
> > > va:138)
> > >
> > >
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Before I switched to using HttpClient years ago, I used HttpsURLConnection. Not sure how things were going to turn out with HttpClient, I kept that code in place and made it so I could use either API. I had her switch back to using the java.net stuff the other day and the problem appeared to go away. I have configured her system to use the java.net connector again today. Before I make any conclusions I need some more data from her on each connector. However, the previous data seems to indicate that it is only an issue when using HttpClient (originally 4.1.3, now 4.2.3).
Some other things of note:
- She is using JRE 6 Update 13. (There is a long story behind this.)
- The application is launched via Webstart
- The SocketFactory is created with: fact = new SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),verifier);
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Friday, January 18, 2013 6:17 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> Thanks for the explanation.
> > pro-actively evicting expired connections and connections that have
> > been idle
> Seems like just doing the stale check would more reliable, and speed
> does not seem to be a factory (especially with 4.2.3)
>
> I got in configured and distributed to this user and she is still
> having the same problem. Any other ideas? I double checked my code,
> and it looks like everything is correct. (She is not using a proxy.)
>
This can also be genuine network stability issues. Essentially, connection resets are inevitable and robust HTTP services have to be prepared to deal with them (most likely by re-trying idempotent methods automatically and reporting the problem back to the caller otherwise).
Oleg
>
> Method that makes the request (from multiple threads)
> {
> response = httpClient.execute(request);
> }
> Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> {
> HttpParams params = new BasicHttpParams();
> params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> configureProxy(params,protocol);
>
> connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
> }
> private void configureProxy(HttpParams params, String scheme) {
> Proxy proxy = getProxy();
> if (proxy.type() != Proxy.Type.DIRECT) {
> InetSocketAddress address = (InetSocketAddress) proxy.address();
> //SSL or not, we set the proxy up as HTTP
> HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> if (!scheme.equals(SCHEME_HTTP)) {
> //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> }
> }
>
> }
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Thursday, January 17, 2013 7:21 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
>
> Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
>
> > I noticed that there is a stale connection check:
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > Boolean.TRUE);
> >
> > Is this something that I should be using? I am connecting to the same source over and over again.
> >
> > Mark
> >
>
> Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
>
> Oleg
>
>
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Wednesday, January 16, 2013 3:07 PM
> > To: 'HttpClient User Discussion'
> > Subject: Connection Reset errors
> >
> > I have a user getting a lot of Connection Reset errors. I did not
> > think this had do to with HttpClient, so much as other factors on
> > her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> >
> > This user is fine for a while, and then will get the Connection
> > Reset error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> >
> > Thanks,
> > Mark
> >
> > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > <Connector
> > port="5502"
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > server="Unknown"
> > connectionLinger="0"
> > socket.soTimeout="300000"
> > SSLEnabled="true"
> > maxThreads="150"
> > scheme="https"
> > secure="true"
> > clientAuth="false"
> > keystoreFile="-----"
> > keystoreType="PKCS12"
> > keystorePass="-----"
> > sslProtocol="TLS"
> >
> > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_D
> > HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE
> > _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> >
> >
> > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > Message (Connection reset)
> > at java.net.SocketInputStream.read(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > at
> > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.
> > ja
> > va:138)
> >
> >
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > --------------------------------------------------------------------
> > - To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote:
> Thanks for the explanation.
> > pro-actively evicting expired connections and connections that have been idle
> Seems like just doing the stale check would more reliable, and speed does not seem to be a factory (especially with 4.2.3)
>
> I got in configured and distributed to this user and she is still having the same problem. Any other ideas? I double checked my code, and it looks like everything is correct. (She is not using a proxy.)
>
This can also be genuine network stability issues. Essentially,
connection resets are inevitable and robust HTTP services have to be
prepared to deal with them (most likely by re-trying idempotent methods
automatically and reporting the problem back to the caller otherwise).
Oleg
>
> Method that makes the request (from multiple threads)
> {
> response = httpClient.execute(request);
> }
> Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
> {
> HttpParams params = new BasicHttpParams();
> params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
> params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
> params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
> params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
> configureProxy(params,protocol);
>
> connectionManager = new PoolingClientConnectionManager(schemeRegistry);
> connectionManager.setDefaultMaxPerRoute(MAX);
> connectionManager.setMaxTotal(MAX);
>
> httpClient = new DefaultHttpClient(connectionManager, params);
> }
> private void configureProxy(HttpParams params, String scheme) {
> Proxy proxy = getProxy();
> if (proxy.type() != Proxy.Type.DIRECT) {
> InetSocketAddress address = (InetSocketAddress) proxy.address();
> //SSL or not, we set the proxy up as HTTP
> HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
> params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
> LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
> if (!scheme.equals(SCHEME_HTTP)) {
> //If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
> schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
> }
> }
>
> }
>
> -----Original Message-----
> From: Oleg Kalnichevski [mailto:olegk@apache.org]
> Sent: Thursday, January 17, 2013 7:21 AM
> To: HttpClient User Discussion
> Subject: Re: Connection Reset errors
>
> On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
>
> Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
>
> > I noticed that there is a stale connection check:
> > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> > Boolean.TRUE);
> >
> > Is this something that I should be using? I am connecting to the same source over and over again.
> >
> > Mark
> >
>
> Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
>
> Oleg
>
>
> > -----Original Message-----
> > From: Mark Claassen [mailto:mac01@donnell.com]
> > Sent: Wednesday, January 16, 2013 3:07 PM
> > To: 'HttpClient User Discussion'
> > Subject: Connection Reset errors
> >
> > I have a user getting a lot of Connection Reset errors. I did not
> > think this had do to with HttpClient, so much as other factors on her
> > machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
> >
> > This user is fine for a while, and then will get the Connection Reset
> > error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
> >
> > Thanks,
> > Mark
> >
> > We are using Tomcat 7.0.27, and the Connector is configured like this:
> > <Connector
> > port="5502"
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > server="Unknown"
> > connectionLinger="0"
> > socket.soTimeout="300000"
> > SSLEnabled="true"
> > maxThreads="150"
> > scheme="https"
> > secure="true"
> > clientAuth="false"
> > keystoreFile="-----"
> > keystoreType="PKCS12"
> > keystorePass="-----"
> > sslProtocol="TLS"
> >
> > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE
> > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_C
> > BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
> >
> >
> > ---- (1) ---- Throwable - Class (class java.net.SocketException)
> > Message (Connection reset)
> > at java.net.SocketInputStream.read(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> > at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> > at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> > at
> > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.ja
> > va:138)
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
Thanks for the explanation.
> pro-actively evicting expired connections and connections that have been idle
Seems like just doing the stale check would more reliable, and speed does not seem to be a factory (especially with 4.2.3)
I got in configured and distributed to this user and she is still having the same problem. Any other ideas? I double checked my code, and it looks like everything is correct. (She is not using a proxy.)
Method that makes the request (from multiple threads)
{
response = httpClient.execute(request);
}
Bulk of configuration method (MAX is 10, CONNECTION_ESTABLISH_TIMEOUT = 10)
{
HttpParams params = new BasicHttpParams();
params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS));
params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, CONNECTION_ESTABLISH_TIMEOUT * 1000);
params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, DEFAULT_HTTP_SOCKET_BUFFER_SIZE);
params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
configureProxy(params,protocol);
connectionManager = new PoolingClientConnectionManager(schemeRegistry);
connectionManager.setDefaultMaxPerRoute(MAX);
connectionManager.setMaxTotal(MAX);
httpClient = new DefaultHttpClient(connectionManager, params);
}
private void configureProxy(HttpParams params, String scheme) {
Proxy proxy = getProxy();
if (proxy.type() != Proxy.Type.DIRECT) {
InetSocketAddress address = (InetSocketAddress) proxy.address();
//SSL or not, we set the proxy up as HTTP
HttpHost httpProxy = new HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP);
params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy);
LogManager.log("Setting proxy in HostConfig to " + address, Level.INFO);
if (!scheme.equals(SCHEME_HTTP)) {
//If the scheme is HTTPS, we need to register an HTTP scheme for the proxy.
schemeRegistry.register(new Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory()));
}
}
}
-----Original Message-----
From: Oleg Kalnichevski [mailto:olegk@apache.org]
Sent: Thursday, January 17, 2013 7:21 AM
To: HttpClient User Discussion
Subject: Re: Connection Reset errors
On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
Basically 'connection reset' errors are a direct result of a general limitation of the blocking I/O model: when not blocked in an I/O operation there is no way for the socket to get notified of the opposite endpoint terminating the connection. While kept alive in the pool blocking HTTP connections can get stale. However, the only way to find it out is to try to attempt an I/O operation on such connection.
> I noticed that there is a stale connection check:
> params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK,
> Boolean.TRUE);
>
> Is this something that I should be using? I am connecting to the same source over and over again.
>
> Mark
>
Yes, this is what you should be using. Another alternative would be pro-actively evicting expired connections and connections that have been idle longer than a given period of time.
Oleg
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Wednesday, January 16, 2013 3:07 PM
> To: 'HttpClient User Discussion'
> Subject: Connection Reset errors
>
> I have a user getting a lot of Connection Reset errors. I did not
> think this had do to with HttpClient, so much as other factors on her
> machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
>
> This user is fine for a while, and then will get the Connection Reset
> error. She then just retries the request, and it works. Any tips on how to resolve this would be greatly appreciated.
>
> Thanks,
> Mark
>
> We are using Tomcat 7.0.27, and the Connector is configured like this:
> <Connector
> port="5502"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> server="Unknown"
> connectionLinger="0"
> socket.soTimeout="300000"
> SSLEnabled="true"
> maxThreads="150"
> scheme="https"
> secure="true"
> clientAuth="false"
> keystoreFile="-----"
> keystoreType="PKCS12"
> keystorePass="-----"
> sslProtocol="TLS"
>
> ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE
> _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_C
> BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
>
>
> ---- (1) ---- Throwable - Class (class java.net.SocketException)
> Message (Connection reset)
> at java.net.SocketInputStream.read(Unknown Source)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> at
> org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.ja
> va:138)
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Connection Reset errors
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote:
Basically 'connection reset' errors are a direct result of a general
limitation of the blocking I/O model: when not blocked in an I/O
operation there is no way for the socket to get notified of the opposite
endpoint terminating the connection. While kept alive in the pool
blocking HTTP connections can get stale. However, the only way to find
it out is to try to attempt an I/O operation on such connection.
> I noticed that there is a stale connection check:
> params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
>
> Is this something that I should be using? I am connecting to the same source over and over again.
>
> Mark
>
Yes, this is what you should be using. Another alternative would be
pro-actively evicting expired connections and connections that have been
idle longer than a given period of time.
Oleg
> -----Original Message-----
> From: Mark Claassen [mailto:mac01@donnell.com]
> Sent: Wednesday, January 16, 2013 3:07 PM
> To: 'HttpClient User Discussion'
> Subject: Connection Reset errors
>
> I have a user getting a lot of Connection Reset errors. I did not think this had do to with HttpClient, so much as other factors on
> her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient
> before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
>
> This user is fine for a while, and then will get the Connection Reset error. She then just retries the request, and it works. Any
> tips on how to resolve this would be greatly appreciated.
>
> Thanks,
> Mark
>
> We are using Tomcat 7.0.27, and the Connector is configured like this:
> <Connector
> port="5502"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> server="Unknown"
> connectionLinger="0"
> socket.soTimeout="300000"
> SSLEnabled="true"
> maxThreads="150"
> scheme="https"
> secure="true"
> clientAuth="false"
> keystoreFile="-----"
> keystoreType="PKCS12"
> keystorePass="-----"
> sslProtocol="TLS"
>
> ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
> _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
>
>
> ---- (1) ---- Throwable - Class (class java.net.SocketException)
> Message (Connection reset)
> at java.net.SocketInputStream.read(Unknown Source)
> at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
> at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
> at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
> at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
> at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
> at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
> at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
> at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: Connection Reset errors
Posted by Mark Claassen <ma...@donnell.com>.
I noticed that there is a stale connection check:
params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, Boolean.TRUE);
Is this something that I should be using? I am connecting to the same source over and over again.
Mark
-----Original Message-----
From: Mark Claassen [mailto:mac01@donnell.com]
Sent: Wednesday, January 16, 2013 3:07 PM
To: 'HttpClient User Discussion'
Subject: Connection Reset errors
I have a user getting a lot of Connection Reset errors. I did not think this had do to with HttpClient, so much as other factors on
her machine and the server. However, I did a bit of searching and see that people have commented on this error with HttpClient
before. I just upgraded to HttpClient 4.2.3 to see if that would help at all, and it didn't.
This user is fine for a while, and then will get the Connection Reset error. She then just retries the request, and it works. Any
tips on how to resolve this would be greatly appreciated.
Thanks,
Mark
We are using Tomcat 7.0.27, and the Connector is configured like this:
<Connector
port="5502"
protocol="org.apache.coyote.http11.Http11NioProtocol"
server="Unknown"
connectionLinger="0"
socket.soTimeout="300000"
SSLEnabled="true"
maxThreads="150"
scheme="https"
secure="true"
clientAuth="false"
keystoreFile="-----"
keystoreType="PKCS12"
keystorePass="-----"
sslProtocol="TLS"
ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES
_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/>
---- (1) ---- Throwable - Class (class java.net.SocketException)
Message (Connection reset)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177)
at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org