You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Karl Wright <da...@gmail.com> on 2013/01/22 15:14:19 UTC
Strange behavior - does this sound familiar to anyone?
Hi all,
For the ManifoldCF release, we've been having problems with our Solr
connector, which uses Solrj which in turn uses
HttpComponents/HttpClient to connect to Solr.
The behavior we've been seeing is that when multiple document uploads
to Solr are taking place at the same time, we see SocketExceptions
that we shouldn't be seeing. These are of the general "broken pipe"
variety, and seem to be consistent with someone somewhere closing a
socket that should not be getting closed. It is furthermore not clear
whether the problem is on the client side or on the server side.
On the client side, we have a connection pool and DefaultHttpClient
instance per thread. I've reviewed all the code we use to set up the
DefaultHttpClient and also the SolrJ code that performs the actual
upload. While I can see where there might be issues (such as unclosed
connections) under some usage scenarios, not only are we avoiding any
of these questionable usage scenarios, but also we are never sharing
HttpClient objects among threads. This leads me to believe that the
problem is on the server side. However, this is difficult for my
colleagues to accept, so I wish to ask the question: is it even
possible for HttpClient to close sockets in a cross-thread manner?
Also, FWIW, I believe that this issue may have been around a fairly
long time, but has been masked by automatic retries in the past.
Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by Karl Wright <da...@gmail.com>.
Thanks, Oleg. I was already aware of that. Architecturally it should
not be possible to share HttpClient objects among threads in this
situation.
Karl
On Wed, Jan 23, 2013 at 5:56 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Tue, 2013-01-22 at 09:14 -0500, Karl Wright wrote:
>> Hi all,
>>
>> For the ManifoldCF release, we've been having problems with our Solr
>> connector, which uses Solrj which in turn uses
>> HttpComponents/HttpClient to connect to Solr.
>>
>> The behavior we've been seeing is that when multiple document uploads
>> to Solr are taking place at the same time, we see SocketExceptions
>> that we shouldn't be seeing. These are of the general "broken pipe"
>> variety, and seem to be consistent with someone somewhere closing a
>> socket that should not be getting closed. It is furthermore not clear
>> whether the problem is on the client side or on the server side.
>>
>> On the client side, we have a connection pool and DefaultHttpClient
>> instance per thread. I've reviewed all the code we use to set up the
>> DefaultHttpClient and also the SolrJ code that performs the actual
>> upload. While I can see where there might be issues (such as unclosed
>> connections) under some usage scenarios, not only are we avoiding any
>> of these questionable usage scenarios, but also we are never sharing
>> HttpClient objects among threads. This leads me to believe that the
>> problem is on the server side. However, this is difficult for my
>> colleagues to accept, so I wish to ask the question: is it even
>> possible for HttpClient to close sockets in a cross-thread manner?
>>
>
> Http connections created by HttpClient are not thread safe (with an
> exception of #shutdown method). HttpClient takes special precautions to
> make sure connections cannot end up being accessed by multiple threads.
> Under normal circumstances client HTTP connection should never be used
> by more than one thread.
>
> I am sorry I cannot be of any better help.
>
> Oleg
>
>> Also, FWIW, I believe that this issue may have been around a fairly
>> long time, but has been masked by automatic retries in the past.
>>
>> Karl
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2013-01-22 at 09:14 -0500, Karl Wright wrote:
> Hi all,
>
> For the ManifoldCF release, we've been having problems with our Solr
> connector, which uses Solrj which in turn uses
> HttpComponents/HttpClient to connect to Solr.
>
> The behavior we've been seeing is that when multiple document uploads
> to Solr are taking place at the same time, we see SocketExceptions
> that we shouldn't be seeing. These are of the general "broken pipe"
> variety, and seem to be consistent with someone somewhere closing a
> socket that should not be getting closed. It is furthermore not clear
> whether the problem is on the client side or on the server side.
>
> On the client side, we have a connection pool and DefaultHttpClient
> instance per thread. I've reviewed all the code we use to set up the
> DefaultHttpClient and also the SolrJ code that performs the actual
> upload. While I can see where there might be issues (such as unclosed
> connections) under some usage scenarios, not only are we avoiding any
> of these questionable usage scenarios, but also we are never sharing
> HttpClient objects among threads. This leads me to believe that the
> problem is on the server side. However, this is difficult for my
> colleagues to accept, so I wish to ask the question: is it even
> possible for HttpClient to close sockets in a cross-thread manner?
>
Http connections created by HttpClient are not thread safe (with an
exception of #shutdown method). HttpClient takes special precautions to
make sure connections cannot end up being accessed by multiple threads.
Under normal circumstances client HTTP connection should never be used
by more than one thread.
I am sorry I cannot be of any better help.
Oleg
> Also, FWIW, I believe that this issue may have been around a fairly
> long time, but has been masked by automatic retries in the past.
>
> Karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by "Asankha C. Perera" <as...@apache.org>.
Hi Karl
You can trace traffic over the "lo" interface on Ubuntu - same with
Wireshark
e.g. sudo tcpdump -i lo -X -n -s 4096 port 8080
regards
asankha
On 01/23/2013 06:08 PM, Karl Wright wrote:
> Ah, but there's a problem. The connection in question is going via
> localhost, and this is Ubuntu or MacOSX, so tcpdump won't see the
> packets. For Windows I'd have to install the Microsoft Loopback
> Adapter which brings its own set of problems. Darn.
>
> Karl
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by Karl Wright <da...@gmail.com>.
Ah, but there's a problem. The connection in question is going via
localhost, and this is Ubuntu or MacOSX, so tcpdump won't see the
packets. For Windows I'd have to install the Microsoft Loopback
Adapter which brings its own set of problems. Darn.
Karl
On Wed, Jan 23, 2013 at 6:47 AM, Karl Wright <da...@gmail.com> wrote:
> We've already looked at wire debugging. See CONNECTORS-616. But with
> Wireshark it is true we could prove who is actually doing the close.
> Good suggestion.
>
> Karl
>
> On Wed, Jan 23, 2013 at 6:05 AM, Asankha C. Perera <as...@apache.org> wrote:
>> If this is not related to load, you could collect a sizable TCP / Wireshark
>> dump to investigate why one party is closing its side of the TCP connection
>>
>> regards
>> asankha
>>
>>
>> On 01/22/2013 10:08 PM, Karl Wright wrote:
>>>
>>> This happens both under Tomcat and under Jetty. It happens sometimes
>>> when only two documents are being indexed at the same time, so it is
>>> not actually related to "load". It also seems to occur during
>>> connection setup rather than when there are established connections.
>>>
>>> Karl
>>
>>
>> --
>> Asankha C. Perera
>> AdroitLogic, http://adroitlogic.org
>>
>> http://esbmagic.blogspot.com
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by Karl Wright <da...@gmail.com>.
We've already looked at wire debugging. See CONNECTORS-616. But with
Wireshark it is true we could prove who is actually doing the close.
Good suggestion.
Karl
On Wed, Jan 23, 2013 at 6:05 AM, Asankha C. Perera <as...@apache.org> wrote:
> If this is not related to load, you could collect a sizable TCP / Wireshark
> dump to investigate why one party is closing its side of the TCP connection
>
> regards
> asankha
>
>
> On 01/22/2013 10:08 PM, Karl Wright wrote:
>>
>> This happens both under Tomcat and under Jetty. It happens sometimes
>> when only two documents are being indexed at the same time, so it is
>> not actually related to "load". It also seems to occur during
>> connection setup rather than when there are established connections.
>>
>> Karl
>
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by "Asankha C. Perera" <as...@apache.org>.
If this is not related to load, you could collect a sizable TCP /
Wireshark dump to investigate why one party is closing its side of the
TCP connection
regards
asankha
On 01/22/2013 10:08 PM, Karl Wright wrote:
> This happens both under Tomcat and under Jetty. It happens sometimes
> when only two documents are being indexed at the same time, so it is
> not actually related to "load". It also seems to occur during
> connection setup rather than when there are established connections.
>
> Karl
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by Karl Wright <da...@gmail.com>.
This happens both under Tomcat and under Jetty. It happens sometimes
when only two documents are being indexed at the same time, so it is
not actually related to "load". It also seems to occur during
connection setup rather than when there are established connections.
Karl
On Tue, Jan 22, 2013 at 10:41 AM, Asankha C. Perera <as...@apache.org> wrote:
> Hi Karl
>
> Is Solr hosted on Tomcat? and is Tomcat under any load / or properly tuned
> to handle such load? How frequently does this occur?
>
> regards
> asankha
>
>
> On 01/22/2013 07:44 PM, Karl Wright wrote:
>>
>> Hi all,
>>
>> For the ManifoldCF release, we've been having problems with our Solr
>> connector, which uses Solrj which in turn uses
>> HttpComponents/HttpClient to connect to Solr.
>>
>> The behavior we've been seeing is that when multiple document uploads
>> to Solr are taking place at the same time, we see SocketExceptions
>> that we shouldn't be seeing. These are of the general "broken pipe"
>> variety, and seem to be consistent with someone somewhere closing a
>> socket that should not be getting closed. It is furthermore not clear
>> whether the problem is on the client side or on the server side.
>>
>> On the client side, we have a connection pool and DefaultHttpClient
>> instance per thread. I've reviewed all the code we use to set up the
>> DefaultHttpClient and also the SolrJ code that performs the actual
>> upload. While I can see where there might be issues (such as unclosed
>> connections) under some usage scenarios, not only are we avoiding any
>> of these questionable usage scenarios, but also we are never sharing
>> HttpClient objects among threads. This leads me to believe that the
>> problem is on the server side. However, this is difficult for my
>> colleagues to accept, so I wish to ask the question: is it even
>> possible for HttpClient to close sockets in a cross-thread manner?
>>
>> Also, FWIW, I believe that this issue may have been around a fairly
>> long time, but has been masked by automatic retries in the past.
>>
>> Karl
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
>> For additional commands, e-mail: dev-help@hc.apache.org
>>
>>
>
>
> --
> Asankha C. Perera
> AdroitLogic, http://adroitlogic.org
>
> http://esbmagic.blogspot.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org
Re: Strange behavior - does this sound familiar to anyone?
Posted by "Asankha C. Perera" <as...@apache.org>.
Hi Karl
Is Solr hosted on Tomcat? and is Tomcat under any load / or properly
tuned to handle such load? How frequently does this occur?
regards
asankha
On 01/22/2013 07:44 PM, Karl Wright wrote:
> Hi all,
>
> For the ManifoldCF release, we've been having problems with our Solr
> connector, which uses Solrj which in turn uses
> HttpComponents/HttpClient to connect to Solr.
>
> The behavior we've been seeing is that when multiple document uploads
> to Solr are taking place at the same time, we see SocketExceptions
> that we shouldn't be seeing. These are of the general "broken pipe"
> variety, and seem to be consistent with someone somewhere closing a
> socket that should not be getting closed. It is furthermore not clear
> whether the problem is on the client side or on the server side.
>
> On the client side, we have a connection pool and DefaultHttpClient
> instance per thread. I've reviewed all the code we use to set up the
> DefaultHttpClient and also the SolrJ code that performs the actual
> upload. While I can see where there might be issues (such as unclosed
> connections) under some usage scenarios, not only are we avoiding any
> of these questionable usage scenarios, but also we are never sharing
> HttpClient objects among threads. This leads me to believe that the
> problem is on the server side. However, this is difficult for my
> colleagues to accept, so I wish to ask the question: is it even
> possible for HttpClient to close sockets in a cross-thread manner?
>
> Also, FWIW, I believe that this issue may have been around a fairly
> long time, but has been masked by automatic retries in the past.
>
> Karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
> For additional commands, e-mail: dev-help@hc.apache.org
>
>
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@hc.apache.org
For additional commands, e-mail: dev-help@hc.apache.org