You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Dmitry Kan <so...@gmail.com> on 2013/05/02 12:57:45 UTC
socket write error
Hi guys!
We have solr router and shards. I see this in jetty log on the router:
May 02, 2013 1:30:22 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.SocketException) caught when processing
request: Connection reset by peer: socket write error
and then:
May 02, 2013 1:30:22 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
followed by exception about Internal Server Error
any ideas why this happens?
We run 80+ shards distributed across several servers. Router runs on its
own node.
Is there anything in particular I should be looking into wrt ubuntu socket
settings? Is this a known issue for solr's distributed search from the past?
Thanks,
Dmitry
Re: socket write error
Posted by Dmitry Kan <so...@gmail.com>.
After some more debugging I have found out, that one of the requests had a
size of 4,4MB. The default maxPostSize in tomcat6 is 2MB (
http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html).
Changing that to 10MB has greatly improved situation on the solr side.
Dmitry
On Fri, May 3, 2013 at 9:55 AM, Dmitry Kan <so...@gmail.com> wrote:
> Digging in further, found this in HttpCommComponent class:
>
> [code]
> static {
> MultiThreadedHttpConnectionManager mgr = new
> MultiThreadedHttpConnectionManager();
> mgr.getParams().setDefaultMaxConnectionsPerHost(20);
> mgr.getParams().setMaxTotalConnections(10000);
> mgr.getParams().setConnectionTimeout(SearchHandler.connectionTimeout);
> mgr.getParams().setSoTimeout(SearchHandler.soTimeout);
> // mgr.getParams().setStaleCheckingEnabled(false);
> client = new HttpClient(mgr);
> }
> [/code]
>
> Could the value set by setDefaultMaxConnectionsPerHost(20) be to small for
> 80+ shards returning results to the router?
>
> Dmitry
>
>
>
> On Fri, May 3, 2013 at 6:50 AM, Dmitry Kan <so...@gmail.com> wrote:
>
>> Hi, thanks.
>>
>> Solr 3.4.
>> There is POST request everywhere, between client and router, router and
>> shards.
>>
>> Do you do faceting across all shards? How many documents approx you have?
>> On 2 May 2013 22:02, "Patanachai Tangchaisin" <
>> patanachai.tangchaisin@wizecommerce.com> wrote:
>>
>>> Hi,
>>>
>>> First, which version of Solr are you using?
>>>
>>> I also has 60 shards+ on Solr 4.2.1 and it doesn't seems to be a problem
>>> for me.
>>>
>>> - Make sure you use POST to send a query to Solr.
>>> - 'connection reset by peer' from client can indicate that there is
>>> something wrong with server e.g. server closes a connection etc.
>>>
>>> --
>>> Patanachai
>>>
>>> On 05/02/2013 05:05 AM, Dmitry Kan wrote:
>>>
>>>> After some searching around, I see this:
>>>>
>>>> http://search-lucene.com/m/**ErEZUl7P5f2/%2522socket+write+**
>>>> error%2522&subj=Long+list+of+**shards+breaks+solrj+query<http://search-lucene.com/m/ErEZUl7P5f2/%2522socket+write+error%2522&subj=Long+list+of+shards+breaks+solrj+query>
>>>>
>>>> Seems like this has happened in the past with large amount of shards.
>>>>
>>>> To make it clear: the distributed search works with 20 shards.
>>>>
>>>>
>>>> On Thu, May 2, 2013 at 1:57 PM, Dmitry Kan <so...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi guys!
>>>>>
>>>>> We have solr router and shards. I see this in jetty log on the router:
>>>>>
>>>>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.**
>>>>> HttpMethodDirector
>>>>> executeWithRetry
>>>>> INFO: I/O exception (java.net.SocketException) caught when processing
>>>>> request: Connection reset by peer: socket write error
>>>>>
>>>>> and then:
>>>>>
>>>>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.**
>>>>> HttpMethodDirector
>>>>> executeWithRetry
>>>>> INFO: Retrying request
>>>>>
>>>>> followed by exception about Internal Server Error
>>>>>
>>>>> any ideas why this happens?
>>>>>
>>>>> We run 80+ shards distributed across several servers. Router runs on
>>>>> its
>>>>> own node.
>>>>>
>>>>> Is there anything in particular I should be looking into wrt ubuntu
>>>>> socket
>>>>> settings? Is this a known issue for solr's distributed search from the
>>>>> past?
>>>>>
>>>>> Thanks,
>>>>> Dmitry
>>>>>
>>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> ======================
>>> This email message and any attachments are for the exclusive use of the
>>> intended recipient(s) and may contain confidential and privileged
>>> information. Any unauthorized review, use, disclosure or distribution is
>>> prohibited. If you are not the intended recipient, please contact the
>>> sender by reply email and destroy all copies of the original message along
>>> with any attachments, from your computer system. If you are the intended
>>> recipient, please be advised that the content of this message is subject to
>>> access, review and disclosure by the sender's Email System Administrator.
>>>
>>>
>
Re: socket write error
Posted by Dmitry Kan <so...@gmail.com>.
Digging in further, found this in HttpCommComponent class:
[code]
static {
MultiThreadedHttpConnectionManager mgr = new
MultiThreadedHttpConnectionManager();
mgr.getParams().setDefaultMaxConnectionsPerHost(20);
mgr.getParams().setMaxTotalConnections(10000);
mgr.getParams().setConnectionTimeout(SearchHandler.connectionTimeout);
mgr.getParams().setSoTimeout(SearchHandler.soTimeout);
// mgr.getParams().setStaleCheckingEnabled(false);
client = new HttpClient(mgr);
}
[/code]
Could the value set by setDefaultMaxConnectionsPerHost(20) be to small for
80+ shards returning results to the router?
Dmitry
On Fri, May 3, 2013 at 6:50 AM, Dmitry Kan <so...@gmail.com> wrote:
> Hi, thanks.
>
> Solr 3.4.
> There is POST request everywhere, between client and router, router and
> shards.
>
> Do you do faceting across all shards? How many documents approx you have?
> On 2 May 2013 22:02, "Patanachai Tangchaisin" <
> patanachai.tangchaisin@wizecommerce.com> wrote:
>
>> Hi,
>>
>> First, which version of Solr are you using?
>>
>> I also has 60 shards+ on Solr 4.2.1 and it doesn't seems to be a problem
>> for me.
>>
>> - Make sure you use POST to send a query to Solr.
>> - 'connection reset by peer' from client can indicate that there is
>> something wrong with server e.g. server closes a connection etc.
>>
>> --
>> Patanachai
>>
>> On 05/02/2013 05:05 AM, Dmitry Kan wrote:
>>
>>> After some searching around, I see this:
>>>
>>> http://search-lucene.com/m/**ErEZUl7P5f2/%2522socket+write+**
>>> error%2522&subj=Long+list+of+**shards+breaks+solrj+query<http://search-lucene.com/m/ErEZUl7P5f2/%2522socket+write+error%2522&subj=Long+list+of+shards+breaks+solrj+query>
>>>
>>> Seems like this has happened in the past with large amount of shards.
>>>
>>> To make it clear: the distributed search works with 20 shards.
>>>
>>>
>>> On Thu, May 2, 2013 at 1:57 PM, Dmitry Kan <so...@gmail.com> wrote:
>>>
>>> Hi guys!
>>>>
>>>> We have solr router and shards. I see this in jetty log on the router:
>>>>
>>>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.**
>>>> HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.SocketException) caught when processing
>>>> request: Connection reset by peer: socket write error
>>>>
>>>> and then:
>>>>
>>>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.**
>>>> HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>>
>>>> followed by exception about Internal Server Error
>>>>
>>>> any ideas why this happens?
>>>>
>>>> We run 80+ shards distributed across several servers. Router runs on its
>>>> own node.
>>>>
>>>> Is there anything in particular I should be looking into wrt ubuntu
>>>> socket
>>>> settings? Is this a known issue for solr's distributed search from the
>>>> past?
>>>>
>>>> Thanks,
>>>> Dmitry
>>>>
>>>>
>>
>> CONFIDENTIALITY NOTICE
>> ======================
>> This email message and any attachments are for the exclusive use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by reply email and destroy all copies of the original message along
>> with any attachments, from your computer system. If you are the intended
>> recipient, please be advised that the content of this message is subject to
>> access, review and disclosure by the sender's Email System Administrator.
>>
>>
Re: socket write error
Posted by Dmitry Kan <so...@gmail.com>.
Hi, thanks.
Solr 3.4.
There is POST request everywhere, between client and router, router and
shards.
Do you do faceting across all shards? How many documents approx you have?
On 2 May 2013 22:02, "Patanachai Tangchaisin" <
patanachai.tangchaisin@wizecommerce.com> wrote:
> Hi,
>
> First, which version of Solr are you using?
>
> I also has 60 shards+ on Solr 4.2.1 and it doesn't seems to be a problem
> for me.
>
> - Make sure you use POST to send a query to Solr.
> - 'connection reset by peer' from client can indicate that there is
> something wrong with server e.g. server closes a connection etc.
>
> --
> Patanachai
>
> On 05/02/2013 05:05 AM, Dmitry Kan wrote:
>
>> After some searching around, I see this:
>>
>> http://search-lucene.com/m/**ErEZUl7P5f2/%2522socket+write+**
>> error%2522&subj=Long+list+of+**shards+breaks+solrj+query<http://search-lucene.com/m/ErEZUl7P5f2/%2522socket+write+error%2522&subj=Long+list+of+shards+breaks+solrj+query>
>>
>> Seems like this has happened in the past with large amount of shards.
>>
>> To make it clear: the distributed search works with 20 shards.
>>
>>
>> On Thu, May 2, 2013 at 1:57 PM, Dmitry Kan <so...@gmail.com> wrote:
>>
>> Hi guys!
>>>
>>> We have solr router and shards. I see this in jetty log on the router:
>>>
>>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.**
>>> HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.SocketException) caught when processing
>>> request: Connection reset by peer: socket write error
>>>
>>> and then:
>>>
>>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.**
>>> HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>>
>>> followed by exception about Internal Server Error
>>>
>>> any ideas why this happens?
>>>
>>> We run 80+ shards distributed across several servers. Router runs on its
>>> own node.
>>>
>>> Is there anything in particular I should be looking into wrt ubuntu
>>> socket
>>> settings? Is this a known issue for solr's distributed search from the
>>> past?
>>>
>>> Thanks,
>>> Dmitry
>>>
>>>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
>
Re: socket write error
Posted by Patanachai Tangchaisin <pa...@wizecommerce.com>.
Hi,
First, which version of Solr are you using?
I also has 60 shards+ on Solr 4.2.1 and it doesn't seems to be a problem
for me.
- Make sure you use POST to send a query to Solr.
- 'connection reset by peer' from client can indicate that there is
something wrong with server e.g. server closes a connection etc.
--
Patanachai
On 05/02/2013 05:05 AM, Dmitry Kan wrote:
> After some searching around, I see this:
>
> http://search-lucene.com/m/ErEZUl7P5f2/%2522socket+write+error%2522&subj=Long+list+of+shards+breaks+solrj+query
>
> Seems like this has happened in the past with large amount of shards.
>
> To make it clear: the distributed search works with 20 shards.
>
>
> On Thu, May 2, 2013 at 1:57 PM, Dmitry Kan <so...@gmail.com> wrote:
>
>> Hi guys!
>>
>> We have solr router and shards. I see this in jetty log on the router:
>>
>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.SocketException) caught when processing
>> request: Connection reset by peer: socket write error
>>
>> and then:
>>
>> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>>
>> followed by exception about Internal Server Error
>>
>> any ideas why this happens?
>>
>> We run 80+ shards distributed across several servers. Router runs on its
>> own node.
>>
>> Is there anything in particular I should be looking into wrt ubuntu socket
>> settings? Is this a known issue for solr's distributed search from the past?
>>
>> Thanks,
>> Dmitry
>>
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: socket write error
Posted by Dmitry Kan <so...@gmail.com>.
After some searching around, I see this:
http://search-lucene.com/m/ErEZUl7P5f2/%2522socket+write+error%2522&subj=Long+list+of+shards+breaks+solrj+query
Seems like this has happened in the past with large amount of shards.
To make it clear: the distributed search works with 20 shards.
On Thu, May 2, 2013 at 1:57 PM, Dmitry Kan <so...@gmail.com> wrote:
> Hi guys!
>
> We have solr router and shards. I see this in jetty log on the router:
>
> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.SocketException) caught when processing
> request: Connection reset by peer: socket write error
>
> and then:
>
> May 02, 2013 1:30:22 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
>
> followed by exception about Internal Server Error
>
> any ideas why this happens?
>
> We run 80+ shards distributed across several servers. Router runs on its
> own node.
>
> Is there anything in particular I should be looking into wrt ubuntu socket
> settings? Is this a known issue for solr's distributed search from the past?
>
> Thanks,
> Dmitry
>