You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Sebastiano Vigna <vi...@di.unimi.it> on 2013/09/25 09:05:05 UTC
Client stuck
We are running 1000 threads doing downloads using HttpClient 4.3. When we try to shut down the system, around 40 threads appear to be stuck inside an HttpClient call (see stack traces). We were expecting that at some point a timeout would occur, but this doesn't happen.
Any suggestion?
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
Ciao,
seba
"FetchingThread-901" daemon prio=10 tid=0x00007fadf033d800 nid=0x461e runnable [0x00007fab9d08f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:195)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:174)
at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:198)
at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:287)
at org.apache.http.impl.execchain.ResponseEntityWrapper.streamClosed(ResponseEntityWrapper.java:120)
at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:227)
at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174)
at org.apache.http.util.EntityUtils.consume(EntityUtils.java:88)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:240)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:261)
"FetchingThread-498" prio=10 tid=0x00007fb424d6f000 nid=0x39e1 runnable [0x00007fb34cdcc000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:195)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:200)
at org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:103)
at org.apache.http.impl.execchain.ResponseEntityWrapper.streamClosed(ResponseEntityWrapper.java:120)
at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:227)
at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174)
at org.apache.http.util.EntityUtils.consume(EntityUtils.java:88)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:240)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:261)
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Ken Krugler <kk...@transpac.com>.
On Sep 25, 2013, at 3:57am, Sebastiano Vigna wrote:
> On 25 Sep 2013, at 12:21 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> There are probably two most possibilities here: (1) there is a bug in
>> HttpClient and the socket value does not correctly apply (I have tested
>> such scenario on a number of occasions, so I do not find it likely), (2)
>> the target servers keep on sending data either infinitely or at a very
>> slow rate (in both cases the connection never reaches the level of
>> inactivity for socket timeout to fire).
>
>
> No, infinitely is impossible because we truncate after 20M.
>
> public Void handleResponse( HttpResponse response ) throws ClientProtocolException, IOException {
> FetchData.this.response = response;
> final HttpEntity entity = response.getEntity();
>
> if ( entity == null ) LOGGER.warn( "Null entity for URL " + url );
> else {
> wrappedEntity.setEntity( entity );
> truncated = wrappedEntity.copyContent( maxResponseBodyLength );
> if ( truncated ) httpGet.abort();
> }
> return null;
> }} );
>
> wrappedEntity simply copies maxResponseBodyLength bytes and then exits.
>
> It could be infinitely slow rate, but frankly netstat does not report *any* open connection.
>
> Nonetheless, after about four hours, 41 out of 42 connections have exited.
>
> Any suggestion to patch this behaviour? One thing we can do is to track the URLs that have caused such stalling connections.
During large scale web crawling we'd often run into servers that trickled data back, which would create effectively "hung" connections.
Our solution is that we loop while reading available data from an InputStream get get via HttpEntity.getContent().
If the read rate drops below a threshold we abort the request.
It would be cleaner to have this built into an instrumented HttpClient - I haven't looked at http://metrics.codahale.com/manual/httpclient/, but seems interesting.
-- Ken
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 7 Oct 2013, at 6:28 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> What about HttpUriRequest#abort? Which is what I actually meant by
> interrupting the tread.
>
OK. That worked. Thanks!
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2013-10-07 at 18:25 +0200, Sebastiano Vigna wrote:
> On 7 Oct 2013, at 7:22 AM, Sebastiano Vigna <vi...@di.unimi.it> wrote:
>
> > It's definitely reproducible, though, so I'll try with all the logging you suggested--maybe we can catch it.
>
>
> The knowledge I've gathered so far:
>
> - It is useless to interrupt the thread.
> - The only way I found to unlock the thread is to manually kill the connection using killcx.
>
What about HttpUriRequest#abort? Which is what I actually meant by
interrupting the tread.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 7 Oct 2013, at 7:22 AM, Sebastiano Vigna <vi...@di.unimi.it> wrote:
> It's definitely reproducible, though, so I'll try with all the logging you suggested--maybe we can catch it.
The knowledge I've gathered so far:
- It is useless to interrupt the thread.
- The only way I found to unlock the thread is to manually kill the connection using killcx.
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 5 Oct 2013, at 2:32 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>> That is, it's normal for the client to be stuck that way (like in those traces), in spite of connection and socket timeouts?
>>
>
> What do you expect me to say? The threads are stuck in a native read
> method. There is no way of knowing that what exactly is going on inside
> of it. If this behavior is reproducible all I can suggest that you run
Well, I just expect it to be documented. It's a kind of surprise.
It's definitely reproducible, though, so I'll try with all the logging you suggested--maybe we can catch it.
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2013-10-05 at 14:14 +0200, Sebastiano Vigna wrote:
> On 5 Oct 2013, at 1:56 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> >> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
> >>
> >
> > Well, that can certainly be seen as the last resort if everything else
> > fails.
>
>
> That is, it's normal for the client to be stuck that way (like in those traces), in spite of connection and socket timeouts?
>
What do you expect me to say? The threads are stuck in a native read
method. There is no way of knowing that what exactly is going on inside
of it. If this behavior is reproducible all I can suggest that you run
the crawler with wire / context logging on and see if there are any
clues to glean from it. Running the JVN with a kernel level tracing
would be another option.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 5 Oct 2013, at 1:56 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
>>
>
> Well, that can certainly be seen as the last resort if everything else
> fails.
That is, it's normal for the client to be stuck that way (like in those traces), in spite of connection and socket timeouts?
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2013-10-05 at 11:39 +0200, Sebastiano Vigna wrote:
> On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > I do not have any suggestions other than taking note of problematic
> > hosts, monitoring them closely and probably applying more aggressive
> > timeout parameters to those hosts by dynamically reducing socket timeout
> > if throughput drops below a certain limit.
>
>
> We did it, and it solved part of the problems we're having.
>
> The fact is that there are parts of the stack we cannot control. This threads are stuck since ~12h (we have about 30 of the same kinds over 6000 overall fetching threads on 3 machines):
...
> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
>
Well, that can certainly be seen as the last resort if everything else
fails.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 7 Oct 2013, at 7:20 PM, Matt Brown <Ma...@citrix.com> wrote:
> Have you considered using the PoolingClientConnectionManager? I see the BasicHttpClientConnectionManager in the stacktrace.
>
> It wouldn't solve the root issue, but if your application is making requests to the same servers over and over again you might be able to save some handshaking and latency by reusing HTTP connections.
Actually, we're using BasicHttpClientConnectionManager with a DefaultConnectionReuseStrategy and really looks like we're reusing HTTP connections when we reconnect to the same host (we checked the TCP packets).
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Matt Brown <Ma...@citrix.com>.
Have you considered using the PoolingClientConnectionManager? I see the BasicHttpClientConnectionManager in the stacktrace.
It wouldn't solve the root issue, but if your application is making requests to the same servers over and over again you might be able to save some handshaking and latency by reusing HTTP connections.
On Oct 5, 2013, at 5:39 AM, Sebastiano Vigna <vi...@di.unimi.it> wrote:
> On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> I do not have any suggestions other than taking note of problematic
>> hosts, monitoring them closely and probably applying more aggressive
>> timeout parameters to those hosts by dynamically reducing socket timeout
>> if throughput drops below a certain limit.
>
>
> We did it, and it solved part of the problems we're having.
>
> The fact is that there are parts of the stack we cannot control. This threads are stuck since ~12h (we have about 30 of the same kinds over 6000 overall fetching threads on 3 machines):
>
> "FetchingThread-151" prio=10 tid=0x00007f0cc4e8f000 nid=0x256b runnable [0x00007f082dfdd000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
> at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
> at sun.security.ssl.InputRecord.read(InputRecord.java:509)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
> - locked <0x0000000576497a70> (a java.lang.Object)
> at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
> - locked <0x0000000576497a30> (a java.lang.Object)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:262)
> at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
> at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:318)
> at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
> at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
> at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
>
>
>
> "FetchingThread-1856" prio=10 tid=0x00007f4a7db9f000 nid=0x26ef runnable [0x00007f41b8b0e000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
> at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
> at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
>
> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
>
> Ciao,
>
> seba
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> I do not have any suggestions other than taking note of problematic
> hosts, monitoring them closely and probably applying more aggressive
> timeout parameters to those hosts by dynamically reducing socket timeout
> if throughput drops below a certain limit.
We did it, and it solved part of the problems we're having.
The fact is that there are parts of the stack we cannot control. This threads are stuck since ~12h (we have about 30 of the same kinds over 6000 overall fetching threads on 3 machines):
"FetchingThread-151" prio=10 tid=0x00007f0cc4e8f000 nid=0x256b runnable [0x00007f082dfdd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
at sun.security.ssl.InputRecord.read(InputRecord.java:509)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
- locked <0x0000000576497a70> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
- locked <0x0000000576497a30> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:262)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:318)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
"FetchingThread-1856" prio=10 tid=0x00007f4a7db9f000 nid=0x26ef runnable [0x00007f41b8b0e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-09-25 at 12:57 +0200, Sebastiano Vigna wrote:
> On 25 Sep 2013, at 12:21 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > There are probably two most possibilities here: (1) there is a bug in
> > HttpClient and the socket value does not correctly apply (I have tested
> > such scenario on a number of occasions, so I do not find it likely), (2)
> > the target servers keep on sending data either infinitely or at a very
> > slow rate (in both cases the connection never reaches the level of
> > inactivity for socket timeout to fire).
>
>
...
>
> It could be infinitely slow rate, but frankly netstat does not report *any* open connection.
>
> Nonetheless, after about four hours, 41 out of 42 connections have exited.
>
> Any suggestion to patch this behaviour? One thing we can do is to track the URLs that have caused such stalling connections.
>
I do not have any suggestions other than taking note of problematic
hosts, monitoring them closely and probably applying more aggressive
timeout parameters to those hosts by dynamically reducing socket timeout
if throughput drops below a certain limit.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 25 Sep 2013, at 12:21 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> There are probably two most possibilities here: (1) there is a bug in
> HttpClient and the socket value does not correctly apply (I have tested
> such scenario on a number of occasions, so I do not find it likely), (2)
> the target servers keep on sending data either infinitely or at a very
> slow rate (in both cases the connection never reaches the level of
> inactivity for socket timeout to fire).
No, infinitely is impossible because we truncate after 20M.
public Void handleResponse( HttpResponse response ) throws ClientProtocolException, IOException {
FetchData.this.response = response;
final HttpEntity entity = response.getEntity();
if ( entity == null ) LOGGER.warn( "Null entity for URL " + url );
else {
wrappedEntity.setEntity( entity );
truncated = wrappedEntity.copyContent( maxResponseBodyLength );
if ( truncated ) httpGet.abort();
}
return null;
}} );
wrappedEntity simply copies maxResponseBodyLength bytes and then exits.
It could be infinitely slow rate, but frankly netstat does not report *any* open connection.
Nonetheless, after about four hours, 41 out of 42 connections have exited.
Any suggestion to patch this behaviour? One thing we can do is to track the URLs that have caused such stalling connections.
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-09-25 at 10:11 +0200, Sebastiano Vigna wrote:
> On 25 Sep 2013, at 9:55 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > Socket timeout is your best friend.
>
>
> Sorry, I thought that was obvious. This is our default request config:
>
> defaultRequestConfig = RequestConfig.custom()
> .setSocketTimeout( rc.socketTimeout )
> .setConnectTimeout( rc.connectionTimeout )
> .setConnectionRequestTimeout( rc.connectionTimeout )
> .setRedirectsEnabled( false )
> .build();
>
> Both rc.socketTimeout and rc.connectionTimout values are equal to 60000.
>
> We perform our request with
> httpGet.setConfig( defaultRequestConfig );
> httpClient.execute( httpHost, httpGet, new ResponseHandler<Void>() {
> ...
> }
>
> Is there any other socket timeout we should be aware of?
>
There are probably two most possibilities here: (1) there is a bug in
HttpClient and the socket value does not correctly apply (I have tested
such scenario on a number of occasions, so I do not find it likely), (2)
the target servers keep on sending data either infinitely or at a very
slow rate (in both cases the connection never reaches the level of
inactivity for socket timeout to fire).
Oleg
PS: please always respond to the list.
> Ciao,
>
> seba
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2013-09-25 at 09:05 +0200, Sebastiano Vigna wrote:
> We are running 1000 threads doing downloads using HttpClient 4.3. When we try to shut down the system, around 40 threads appear to be stuck inside an HttpClient call (see stack traces). We were expecting that at some point a timeout would occur, but this doesn't happen.
>
> Any suggestion?
>
Socket timeout is your best friend.
Oleg
> java version "1.7.0_40"
> Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
> Java HotSpot(TM) 64-Bit Server VM (build 24.0-b56, mixed mode)
>
> Ciao,
>
> seba
>
> "FetchingThread-901" daemon prio=10 tid=0x00007fadf033d800 nid=0x461e runnable [0x00007fab9d08f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
> at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:195)
> at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:174)
> at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:198)
> at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:287)
> at org.apache.http.impl.execchain.ResponseEntityWrapper.streamClosed(ResponseEntityWrapper.java:120)
> at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:227)
> at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174)
> at org.apache.http.util.EntityUtils.consume(EntityUtils.java:88)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:240)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
> at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
> at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:261)
>
>
> "FetchingThread-498" prio=10 tid=0x00007fb424d6f000 nid=0x39e1 runnable [0x00007fb34cdcc000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
> at org.apache.http.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:195)
> at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178)
> at org.apache.http.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:200)
> at org.apache.http.impl.io.ContentLengthInputStream.close(ContentLengthInputStream.java:103)
> at org.apache.http.impl.execchain.ResponseEntityWrapper.streamClosed(ResponseEntityWrapper.java:120)
> at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:227)
> at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:174)
> at org.apache.http.util.EntityUtils.consume(EntityUtils.java:88)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:240)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
> at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
> at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:261)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org