You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Sebastiano Vigna <vi...@di.unimi.it> on 2013/10/05 11:39:03 UTC
Re: Client stuck
On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> I do not have any suggestions other than taking note of problematic
> hosts, monitoring them closely and probably applying more aggressive
> timeout parameters to those hosts by dynamically reducing socket timeout
> if throughput drops below a certain limit.
We did it, and it solved part of the problems we're having.
The fact is that there are parts of the stack we cannot control. This threads are stuck since ~12h (we have about 30 of the same kinds over 6000 overall fetching threads on 3 machines):
"FetchingThread-151" prio=10 tid=0x00007f0cc4e8f000 nid=0x256b runnable [0x00007f082dfdd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
at sun.security.ssl.InputRecord.read(InputRecord.java:509)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
- locked <0x0000000576497a70> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
- locked <0x0000000576497a30> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:262)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:318)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
"FetchingThread-1856" prio=10 tid=0x00007f4a7db9f000 nid=0x26ef runnable [0x00007f41b8b0e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 7 Oct 2013, at 6:28 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> What about HttpUriRequest#abort? Which is what I actually meant by
> interrupting the tread.
>
OK. That worked. Thanks!
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2013-10-07 at 18:25 +0200, Sebastiano Vigna wrote:
> On 7 Oct 2013, at 7:22 AM, Sebastiano Vigna <vi...@di.unimi.it> wrote:
>
> > It's definitely reproducible, though, so I'll try with all the logging you suggested--maybe we can catch it.
>
>
> The knowledge I've gathered so far:
>
> - It is useless to interrupt the thread.
> - The only way I found to unlock the thread is to manually kill the connection using killcx.
>
What about HttpUriRequest#abort? Which is what I actually meant by
interrupting the tread.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 7 Oct 2013, at 7:22 AM, Sebastiano Vigna <vi...@di.unimi.it> wrote:
> It's definitely reproducible, though, so I'll try with all the logging you suggested--maybe we can catch it.
The knowledge I've gathered so far:
- It is useless to interrupt the thread.
- The only way I found to unlock the thread is to manually kill the connection using killcx.
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 5 Oct 2013, at 2:32 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>> That is, it's normal for the client to be stuck that way (like in those traces), in spite of connection and socket timeouts?
>>
>
> What do you expect me to say? The threads are stuck in a native read
> method. There is no way of knowing that what exactly is going on inside
> of it. If this behavior is reproducible all I can suggest that you run
Well, I just expect it to be documented. It's a kind of surprise.
It's definitely reproducible, though, so I'll try with all the logging you suggested--maybe we can catch it.
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2013-10-05 at 14:14 +0200, Sebastiano Vigna wrote:
> On 5 Oct 2013, at 1:56 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> >> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
> >>
> >
> > Well, that can certainly be seen as the last resort if everything else
> > fails.
>
>
> That is, it's normal for the client to be stuck that way (like in those traces), in spite of connection and socket timeouts?
>
What do you expect me to say? The threads are stuck in a native read
method. There is no way of knowing that what exactly is going on inside
of it. If this behavior is reproducible all I can suggest that you run
the crawler with wire / context logging on and see if there are any
clues to glean from it. Running the JVN with a kernel level tracing
would be another option.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 5 Oct 2013, at 1:56 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
>>
>
> Well, that can certainly be seen as the last resort if everything else
> fails.
That is, it's normal for the client to be stuck that way (like in those traces), in spite of connection and socket timeouts?
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Sat, 2013-10-05 at 11:39 +0200, Sebastiano Vigna wrote:
> On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
> > I do not have any suggestions other than taking note of problematic
> > hosts, monitoring them closely and probably applying more aggressive
> > timeout parameters to those hosts by dynamically reducing socket timeout
> > if throughput drops below a certain limit.
>
>
> We did it, and it solved part of the problems we're having.
>
> The fact is that there are parts of the stack we cannot control. This threads are stuck since ~12h (we have about 30 of the same kinds over 6000 overall fetching threads on 3 machines):
...
> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
>
Well, that can certainly be seen as the last resort if everything else
fails.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Sebastiano Vigna <vi...@di.unimi.it>.
On 7 Oct 2013, at 7:20 PM, Matt Brown <Ma...@citrix.com> wrote:
> Have you considered using the PoolingClientConnectionManager? I see the BasicHttpClientConnectionManager in the stacktrace.
>
> It wouldn't solve the root issue, but if your application is making requests to the same servers over and over again you might be able to save some handshaking and latency by reusing HTTP connections.
Actually, we're using BasicHttpClientConnectionManager with a DefaultConnectionReuseStrategy and really looks like we're reusing HTTP connections when we reconnect to the same host (we checked the TCP packets).
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Client stuck
Posted by Matt Brown <Ma...@citrix.com>.
Have you considered using the PoolingClientConnectionManager? I see the BasicHttpClientConnectionManager in the stacktrace.
It wouldn't solve the root issue, but if your application is making requests to the same servers over and over again you might be able to save some handshaking and latency by reusing HTTP connections.
On Oct 5, 2013, at 5:39 AM, Sebastiano Vigna <vi...@di.unimi.it> wrote:
> On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> I do not have any suggestions other than taking note of problematic
>> hosts, monitoring them closely and probably applying more aggressive
>> timeout parameters to those hosts by dynamically reducing socket timeout
>> if throughput drops below a certain limit.
>
>
> We did it, and it solved part of the problems we're having.
>
> The fact is that there are parts of the stack we cannot control. This threads are stuck since ~12h (we have about 30 of the same kinds over 6000 overall fetching threads on 3 machines):
>
> "FetchingThread-151" prio=10 tid=0x00007f0cc4e8f000 nid=0x256b runnable [0x00007f082dfdd000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
> at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
> at sun.security.ssl.InputRecord.read(InputRecord.java:509)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
> - locked <0x0000000576497a70> (a java.lang.Object)
> at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
> - locked <0x0000000576497a30> (a java.lang.Object)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
> at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
> at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:262)
> at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
> at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:318)
> at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
> at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
> at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
>
>
>
> "FetchingThread-1856" prio=10 tid=0x00007f4a7db9f000 nid=0x26ef runnable [0x00007f41b8b0e000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:152)
> at java.net.SocketInputStream.read(SocketInputStream.java:122)
> at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
> at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
> at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
> at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
> at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
> at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
> at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253)
> at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
> at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
> at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
> at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
> at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
> at it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
>
> Suggestions? Should we give an interrupt after a certain amount of time has elapsed?
>
> Ciao,
>
> seba
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org