You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Vincenzo D'Amore <v....@gmail.com> on 2015/12/23 01:58:09 UTC
How to check when a search exceeds the threshold of timeAllowed parameter
Hi All,
my website is under pressure, there is a big number of concurrent searches.
When the connected users are too many, the searches becomes so slow that in
some cases users have to wait many seconds.
The queue of searches becomes so long that, in same cases, servers are
blocked trying to serve all these requests.
As far as I know because some searches are very expensive, and when many
expensive searches clog the queue server becomes unresponsive.
In order to quickly workaround this herd effect, I have added a
default timeAllowed to 15 seconds, and this seems help a lot.
But during stress tests but I'm unable to understand when and what requests
are affected by timeAllowed parameter.
Just be clear, I have configure timeAllowed parameter in a SolrCloud
environment, given that partial results may be returned (if there are any),
how can I know when this happens? When the timeAllowed parameter trigger a
partial answer?
Best regards,
Vincenzo
--
Vincenzo D'Amore
email: v.damore@gmail.com
skype: free.dev
mobile: +39 349 8513251
Re: How to check when a search exceeds the threshold of timeAllowed parameter
Posted by Jack Krupansky <ja...@gmail.com>.
timeAllowed was designed to handle queries that by themselves consume lots
of resources, not to try to handle situations with large numbers of
requests that starve other requests from accessing CPU and I/O resources.
The usual technique for handling large numbers of requests is replication,
making more copies of the index that can each be searched in parallel.
How long do queries take when the site is operating normally?
Make sure that you have enough system memory to cache the index, otherwise
the machine wish be thrashing with lots of I/O for competing requests.
-- Jack Krupansky
On Tue, Dec 22, 2015 at 8:43 PM, Vincenzo D'Amore <v....@gmail.com>
wrote:
> Well... I can write everything, but really all this just to understand
> when timeAllowed
> parameter trigger a partial answer? I mean, isn't there anything set in the
> response when is partial?
>
> On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood <wu...@wunderwood.org>
> wrote:
>
> > We need to know a LOT more about your site. Number of documents, size of
> > index, frequency of updates, length of queries approximate size of server
> > (CPUs, RAM, type of disk), version of Solr, version of Java, and features
> > you are using (faceting, highlighting, etc.).
> >
> > After that, we’ll have more questions.
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> >
> > > On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore <v....@gmail.com>
> > wrote:
> > >
> > > Hi All,
> > >
> > > my website is under pressure, there is a big number of concurrent
> > searches.
> > > When the connected users are too many, the searches becomes so slow
> that
> > in
> > > some cases users have to wait many seconds.
> > > The queue of searches becomes so long that, in same cases, servers are
> > > blocked trying to serve all these requests.
> > > As far as I know because some searches are very expensive, and when
> many
> > > expensive searches clog the queue server becomes unresponsive.
> > >
> > > In order to quickly workaround this herd effect, I have added a
> > > default timeAllowed to 15 seconds, and this seems help a lot.
> > >
> > > But during stress tests but I'm unable to understand when and what
> > requests
> > > are affected by timeAllowed parameter.
> > >
> > > Just be clear, I have configure timeAllowed parameter in a SolrCloud
> > > environment, given that partial results may be returned (if there are
> > any),
> > > how can I know when this happens? When the timeAllowed parameter
> trigger
> > a
> > > partial answer?
> > >
> > > Best regards,
> > > Vincenzo
> > >
> > >
> > >
> > > --
> > > Vincenzo D'Amore
> > > email: v.damore@gmail.com
> > > skype: free.dev
> > > mobile: +39 349 8513251
> >
> >
>
>
> --
> Vincenzo D'Amore
> email: v.damore@gmail.com
> skype: free.dev
> mobile: +39 349 8513251
>
Re: How to check when a search exceeds the threshold of timeAllowed parameter
Posted by William Bell <bi...@gmail.com>.
Great I must have missed that.
On Wed, Dec 23, 2015 at 9:41 AM, Jeff Wartes <jw...@whitepages.com> wrote:
> Looks like it’ll set partialResults=true on your results if you hit the
> timeout.
>
> https://issues.apache.org/jira/browse/SOLR-502
>
> https://issues.apache.org/jira/browse/SOLR-5986
>
>
>
>
>
>
> On 12/22/15, 5:43 PM, "Vincenzo D'Amore" <v....@gmail.com> wrote:
>
> >Well... I can write everything, but really all this just to understand
> >when timeAllowed
> >parameter trigger a partial answer? I mean, isn't there anything set in
> the
> >response when is partial?
> >
> >On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood <wu...@wunderwood.org>
> >wrote:
> >
> >> We need to know a LOT more about your site. Number of documents, size of
> >> index, frequency of updates, length of queries approximate size of
> server
> >> (CPUs, RAM, type of disk), version of Solr, version of Java, and
> features
> >> you are using (faceting, highlighting, etc.).
> >>
> >> After that, we’ll have more questions.
> >>
> >> wunder
> >> Walter Underwood
> >> wunder@wunderwood.org
> >> http://observer.wunderwood.org/ (my blog)
> >>
> >>
> >> > On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore <v....@gmail.com>
> >> wrote:
> >> >
> >> > Hi All,
> >> >
> >> > my website is under pressure, there is a big number of concurrent
> >> searches.
> >> > When the connected users are too many, the searches becomes so slow
> that
> >> in
> >> > some cases users have to wait many seconds.
> >> > The queue of searches becomes so long that, in same cases, servers are
> >> > blocked trying to serve all these requests.
> >> > As far as I know because some searches are very expensive, and when
> many
> >> > expensive searches clog the queue server becomes unresponsive.
> >> >
> >> > In order to quickly workaround this herd effect, I have added a
> >> > default timeAllowed to 15 seconds, and this seems help a lot.
> >> >
> >> > But during stress tests but I'm unable to understand when and what
> >> requests
> >> > are affected by timeAllowed parameter.
> >> >
> >> > Just be clear, I have configure timeAllowed parameter in a SolrCloud
> >> > environment, given that partial results may be returned (if there are
> >> any),
> >> > how can I know when this happens? When the timeAllowed parameter
> trigger
> >> a
> >> > partial answer?
> >> >
> >> > Best regards,
> >> > Vincenzo
> >> >
> >> >
> >> >
> >> > --
> >> > Vincenzo D'Amore
> >> > email: v.damore@gmail.com
> >> > skype: free.dev
> >> > mobile: +39 349 8513251
> >>
> >>
> >
> >
> >--
> >Vincenzo D'Amore
> >email: v.damore@gmail.com
> >skype: free.dev
> >mobile: +39 349 8513251
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: How to check when a search exceeds the threshold of timeAllowed
parameter
Posted by Jeff Wartes <jw...@whitepages.com>.
Looks like it’ll set partialResults=true on your results if you hit the timeout.
https://issues.apache.org/jira/browse/SOLR-502
https://issues.apache.org/jira/browse/SOLR-5986
On 12/22/15, 5:43 PM, "Vincenzo D'Amore" <v....@gmail.com> wrote:
>Well... I can write everything, but really all this just to understand
>when timeAllowed
>parameter trigger a partial answer? I mean, isn't there anything set in the
>response when is partial?
>
>On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood <wu...@wunderwood.org>
>wrote:
>
>> We need to know a LOT more about your site. Number of documents, size of
>> index, frequency of updates, length of queries approximate size of server
>> (CPUs, RAM, type of disk), version of Solr, version of Java, and features
>> you are using (faceting, highlighting, etc.).
>>
>> After that, we’ll have more questions.
>>
>> wunder
>> Walter Underwood
>> wunder@wunderwood.org
>> http://observer.wunderwood.org/ (my blog)
>>
>>
>> > On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore <v....@gmail.com>
>> wrote:
>> >
>> > Hi All,
>> >
>> > my website is under pressure, there is a big number of concurrent
>> searches.
>> > When the connected users are too many, the searches becomes so slow that
>> in
>> > some cases users have to wait many seconds.
>> > The queue of searches becomes so long that, in same cases, servers are
>> > blocked trying to serve all these requests.
>> > As far as I know because some searches are very expensive, and when many
>> > expensive searches clog the queue server becomes unresponsive.
>> >
>> > In order to quickly workaround this herd effect, I have added a
>> > default timeAllowed to 15 seconds, and this seems help a lot.
>> >
>> > But during stress tests but I'm unable to understand when and what
>> requests
>> > are affected by timeAllowed parameter.
>> >
>> > Just be clear, I have configure timeAllowed parameter in a SolrCloud
>> > environment, given that partial results may be returned (if there are
>> any),
>> > how can I know when this happens? When the timeAllowed parameter trigger
>> a
>> > partial answer?
>> >
>> > Best regards,
>> > Vincenzo
>> >
>> >
>> >
>> > --
>> > Vincenzo D'Amore
>> > email: v.damore@gmail.com
>> > skype: free.dev
>> > mobile: +39 349 8513251
>>
>>
>
>
>--
>Vincenzo D'Amore
>email: v.damore@gmail.com
>skype: free.dev
>mobile: +39 349 8513251
Re: How to check when a search exceeds the threshold of timeAllowed
parameter
Posted by David Santamauro <da...@gmail.com>.
On 12/23/2015 01:42 AM, William Bell wrote:
> I agree that when using timeAllowed in the header info there should be an
> entry that indicates timeAllowed triggered.
If I'm not mistaken, there is
=> partialResults:true
"responseHeader":{ "partialResults":true }
//
> This is the only reason why we have not used timeAllowed. So this is a
> great suggestion. Something like: <int name="timeAllowedStatus">1</int> ??
> That would be great.
>
> <lst name="responseHeader">
> <int name="status">0</int>
> <int name="timeAllowedStatus">1</int>
> <int name="QTime">107</int>
> <lst name="params">
> <str name="q">*:*</str>
> <str name="timeAllowed">1000</str>
> </lst>
> </lst>
>
>
>
> On Tue, Dec 22, 2015 at 6:43 PM, Vincenzo D'Amore <v....@gmail.com>
> wrote:
>
>> Well... I can write everything, but really all this just to understand
>> when timeAllowed
>> parameter trigger a partial answer? I mean, isn't there anything set in the
>> response when is partial?
>>
>> On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood <wu...@wunderwood.org>
>> wrote:
>>
>>> We need to know a LOT more about your site. Number of documents, size of
>>> index, frequency of updates, length of queries approximate size of server
>>> (CPUs, RAM, type of disk), version of Solr, version of Java, and features
>>> you are using (faceting, highlighting, etc.).
>>>
>>> After that, we’ll have more questions.
>>>
>>> wunder
>>> Walter Underwood
>>> wunder@wunderwood.org
>>> http://observer.wunderwood.org/ (my blog)
>>>
>>>
>>>> On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore <v....@gmail.com>
>>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> my website is under pressure, there is a big number of concurrent
>>> searches.
>>>> When the connected users are too many, the searches becomes so slow
>> that
>>> in
>>>> some cases users have to wait many seconds.
>>>> The queue of searches becomes so long that, in same cases, servers are
>>>> blocked trying to serve all these requests.
>>>> As far as I know because some searches are very expensive, and when
>> many
>>>> expensive searches clog the queue server becomes unresponsive.
>>>>
>>>> In order to quickly workaround this herd effect, I have added a
>>>> default timeAllowed to 15 seconds, and this seems help a lot.
>>>>
>>>> But during stress tests but I'm unable to understand when and what
>>> requests
>>>> are affected by timeAllowed parameter.
>>>>
>>>> Just be clear, I have configure timeAllowed parameter in a SolrCloud
>>>> environment, given that partial results may be returned (if there are
>>> any),
>>>> how can I know when this happens? When the timeAllowed parameter
>> trigger
>>> a
>>>> partial answer?
>>>>
>>>> Best regards,
>>>> Vincenzo
>>>>
>>>>
>>>>
>>>> --
>>>> Vincenzo D'Amore
>>>> email: v.damore@gmail.com
>>>> skype: free.dev
>>>> mobile: +39 349 8513251
>>>
>>>
>>
>>
>> --
>> Vincenzo D'Amore
>> email: v.damore@gmail.com
>> skype: free.dev
>> mobile: +39 349 8513251
>>
>
>
>
Re: How to check when a search exceeds the threshold of timeAllowed parameter
Posted by William Bell <bi...@gmail.com>.
I agree that when using timeAllowed in the header info there should be an
entry that indicates timeAllowed triggered.
This is the only reason why we have not used timeAllowed. So this is a
great suggestion. Something like: <int name="timeAllowedStatus">1</int> ??
That would be great.
<lst name="responseHeader">
<int name="status">0</int>
<int name="timeAllowedStatus">1</int>
<int name="QTime">107</int>
<lst name="params">
<str name="q">*:*</str>
<str name="timeAllowed">1000</str>
</lst>
</lst>
On Tue, Dec 22, 2015 at 6:43 PM, Vincenzo D'Amore <v....@gmail.com>
wrote:
> Well... I can write everything, but really all this just to understand
> when timeAllowed
> parameter trigger a partial answer? I mean, isn't there anything set in the
> response when is partial?
>
> On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood <wu...@wunderwood.org>
> wrote:
>
> > We need to know a LOT more about your site. Number of documents, size of
> > index, frequency of updates, length of queries approximate size of server
> > (CPUs, RAM, type of disk), version of Solr, version of Java, and features
> > you are using (faceting, highlighting, etc.).
> >
> > After that, we’ll have more questions.
> >
> > wunder
> > Walter Underwood
> > wunder@wunderwood.org
> > http://observer.wunderwood.org/ (my blog)
> >
> >
> > > On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore <v....@gmail.com>
> > wrote:
> > >
> > > Hi All,
> > >
> > > my website is under pressure, there is a big number of concurrent
> > searches.
> > > When the connected users are too many, the searches becomes so slow
> that
> > in
> > > some cases users have to wait many seconds.
> > > The queue of searches becomes so long that, in same cases, servers are
> > > blocked trying to serve all these requests.
> > > As far as I know because some searches are very expensive, and when
> many
> > > expensive searches clog the queue server becomes unresponsive.
> > >
> > > In order to quickly workaround this herd effect, I have added a
> > > default timeAllowed to 15 seconds, and this seems help a lot.
> > >
> > > But during stress tests but I'm unable to understand when and what
> > requests
> > > are affected by timeAllowed parameter.
> > >
> > > Just be clear, I have configure timeAllowed parameter in a SolrCloud
> > > environment, given that partial results may be returned (if there are
> > any),
> > > how can I know when this happens? When the timeAllowed parameter
> trigger
> > a
> > > partial answer?
> > >
> > > Best regards,
> > > Vincenzo
> > >
> > >
> > >
> > > --
> > > Vincenzo D'Amore
> > > email: v.damore@gmail.com
> > > skype: free.dev
> > > mobile: +39 349 8513251
> >
> >
>
>
> --
> Vincenzo D'Amore
> email: v.damore@gmail.com
> skype: free.dev
> mobile: +39 349 8513251
>
--
Bill Bell
billnbell@gmail.com
cell 720-256-8076
Re: How to check when a search exceeds the threshold of timeAllowed parameter
Posted by Vincenzo D'Amore <v....@gmail.com>.
Well... I can write everything, but really all this just to understand
when timeAllowed
parameter trigger a partial answer? I mean, isn't there anything set in the
response when is partial?
On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood <wu...@wunderwood.org>
wrote:
> We need to know a LOT more about your site. Number of documents, size of
> index, frequency of updates, length of queries approximate size of server
> (CPUs, RAM, type of disk), version of Solr, version of Java, and features
> you are using (faceting, highlighting, etc.).
>
> After that, we’ll have more questions.
>
> wunder
> Walter Underwood
> wunder@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
>
> > On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore <v....@gmail.com>
> wrote:
> >
> > Hi All,
> >
> > my website is under pressure, there is a big number of concurrent
> searches.
> > When the connected users are too many, the searches becomes so slow that
> in
> > some cases users have to wait many seconds.
> > The queue of searches becomes so long that, in same cases, servers are
> > blocked trying to serve all these requests.
> > As far as I know because some searches are very expensive, and when many
> > expensive searches clog the queue server becomes unresponsive.
> >
> > In order to quickly workaround this herd effect, I have added a
> > default timeAllowed to 15 seconds, and this seems help a lot.
> >
> > But during stress tests but I'm unable to understand when and what
> requests
> > are affected by timeAllowed parameter.
> >
> > Just be clear, I have configure timeAllowed parameter in a SolrCloud
> > environment, given that partial results may be returned (if there are
> any),
> > how can I know when this happens? When the timeAllowed parameter trigger
> a
> > partial answer?
> >
> > Best regards,
> > Vincenzo
> >
> >
> >
> > --
> > Vincenzo D'Amore
> > email: v.damore@gmail.com
> > skype: free.dev
> > mobile: +39 349 8513251
>
>
--
Vincenzo D'Amore
email: v.damore@gmail.com
skype: free.dev
mobile: +39 349 8513251
Re: How to check when a search exceeds the threshold of timeAllowed parameter
Posted by Walter Underwood <wu...@wunderwood.org>.
We need to know a LOT more about your site. Number of documents, size of index, frequency of updates, length of queries approximate size of server (CPUs, RAM, type of disk), version of Solr, version of Java, and features you are using (faceting, highlighting, etc.).
After that, we’ll have more questions.
wunder
Walter Underwood
wunder@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore <v....@gmail.com> wrote:
>
> Hi All,
>
> my website is under pressure, there is a big number of concurrent searches.
> When the connected users are too many, the searches becomes so slow that in
> some cases users have to wait many seconds.
> The queue of searches becomes so long that, in same cases, servers are
> blocked trying to serve all these requests.
> As far as I know because some searches are very expensive, and when many
> expensive searches clog the queue server becomes unresponsive.
>
> In order to quickly workaround this herd effect, I have added a
> default timeAllowed to 15 seconds, and this seems help a lot.
>
> But during stress tests but I'm unable to understand when and what requests
> are affected by timeAllowed parameter.
>
> Just be clear, I have configure timeAllowed parameter in a SolrCloud
> environment, given that partial results may be returned (if there are any),
> how can I know when this happens? When the timeAllowed parameter trigger a
> partial answer?
>
> Best regards,
> Vincenzo
>
>
>
> --
> Vincenzo D'Amore
> email: v.damore@gmail.com
> skype: free.dev
> mobile: +39 349 8513251