Shard timeouts on large (1B docs) Solr cluster

[Jay Hill <ja...@gmail.com>]

I'm on a project where we have 1B docs sharded across 20 servers. We're not
in production yet and we're doing load tests now. We're sending load to hit
100qps per server. As the load increases we're seeing query times
sporadically increasing to 10 seconds, 20 seconds, etc. at times. What
we're trying to do is set a shard timeout so that responses longer than 2
seconds are discarded. We can live with less results in these cases. We're
not replicating yet as we want to see how the 20 shards perform first (plus
we're waiting on the massive amount of hardware)

I've tried setting the following config in our default req. handler:
<int name="shard-socket-timeout">2000</int>
<int name="shard-connection-timeout">2000</int>

I've just added these, and am testing now, but this doesn't look promising
either:
<int name="timeAllowed">2000</int>
<bool name="partialResults">true</bool>

Couldn't find much on the wiki about these params - I'm looking for more
details about how these work. I'll be happy to update the wiki with more
details based on the discussion here.

Any details about exactly how I can achieve my goal of timing out and
disregarding queries longer that 2 seconds would be greatly appreciated.

The index is insanely lean - no stored fields, no norms, no stop words,
etc. RAM buffer is 128, and we're using the standard "search" req. handler.
Essentially we're running Solr as a nosql data store, which suits this
project, but we need responses to be no longer than 2 seconds at the max.

Thanks,
-Jay

Re: Shard timeouts on large (1B docs) Solr cluster

[Marc Sturlese <ma...@gmail.com>]

timeAllowed can be used outside distributed search. It is used by the
TimeL¡mitingCollector. When the search time is equal to timeAllowed it will
stop searching and will return the results that could find till then.
This can be a problem when using incremental indexing. Lucene starts
searching from "the bottom" and new docs are inserted on the top, so,
timeAllowed could cause that new docs never appear on the search results.

--
View this message in context: http://lucene.472066.n3.nabble.com/Shard-timeouts-on-large-1B-docs-Solr-cluster-tp3691229p3713263.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Shard timeouts on large (1B docs) Solr cluster

[Jay Hill <ja...@gmail.com>]

i'm changing the params to socketTimeout and connTimeout and will test this
afternoon. client timeout was actually removed today, which helped a bit.

what about the other params, "timeAllowed" and "partialResults". my
expectation was that these were specifically for distributed search,
meaning if a response wasn't received w/in the timeAllowed, and if
partialResults is true, then that shard would not be waited on for results.
is that correct?

thanks,
-jay


On Thu, Jan 26, 2012 at 2:23 PM, Jay Hill <ja...@gmail.com> wrote:

> We're on the trunk:
> 4.0-2011-10-26_08-46-59 1189079 - hudson - 2011-10-26 08:51:47
>
> Client timeouts are set to 4 seconds.
>
> Thanks,
> -Jay
>
>
> On Thu, Jan 26, 2012 at 1:40 PM, Mark Miller <ma...@gmail.com>wrote:
>
>>
>> On Jan 26, 2012, at 1:28 PM, Jay Hill wrote:
>>
>> >
>> > I've tried setting the following config in our default req. handler:
>> > <int name="shard-socket-timeout">2000</int>
>> > <int name="shard-connection-timeout">2000</int>
>> >
>>
>>
>> What version are you using Jay? At least on trunk, I took a look and it
>> appears at some point these where renamed to socketTimeout and connTimeout.
>>
>> What about a timeout on your clients?
>>
>> - Mark Miller
>> lucidimagination.com
>>
>>
>

Re: Shard timeouts on large (1B docs) Solr cluster

[Jay Hill <ja...@gmail.com>]

We're on the trunk:
4.0-2011-10-26_08-46-59 1189079 - hudson - 2011-10-26 08:51:47

Client timeouts are set to 4 seconds.

Thanks,
-Jay

On Thu, Jan 26, 2012 at 1:40 PM, Mark Miller <ma...@gmail.com> wrote:

>
> On Jan 26, 2012, at 1:28 PM, Jay Hill wrote:
>
> >
> > I've tried setting the following config in our default req. handler:
> > <int name="shard-socket-timeout">2000</int>
> > <int name="shard-connection-timeout">2000</int>
> >
>
>
> What version are you using Jay? At least on trunk, I took a look and it
> appears at some point these where renamed to socketTimeout and connTimeout.
>
> What about a timeout on your clients?
>
> - Mark Miller
> lucidimagination.com
>
>

Re: Shard timeouts on large (1B docs) Solr cluster

[Mark Miller <ma...@gmail.com>]

On Jan 26, 2012, at 1:28 PM, Jay Hill wrote:

> 
> I've tried setting the following config in our default req. handler:
> <int name="shard-socket-timeout">2000</int>
> <int name="shard-connection-timeout">2000</int>
> 


What version are you using Jay? At least on trunk, I took a look and it appears at some point these where renamed to socketTimeout and connTimeout.

What about a timeout on your clients?

- Mark Miller
lucidimagination.com