You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Robinson <ma...@gmail.com> on 2020/09/15 13:13:51 UTC
"timeAllowed" param with "numFound" having a count value but doc list
is empty
Hi,
When in a sample query I used "timeAllowed" as low as 10mS, I got value for
"numFound" as say 2000, but no docs were returned. But when I increased the
value for timeAllowed to be in seconds, never got this scenario.
I have 2 qns:-
1. Why does numFound have a value like say 2000 or even 6000 but no
documents actually returned. During document collection is calculation of
numFound done first and doc collection later?. Is doc list empty because,by
the time doc collection started the timeAllowed cut off took effect?
2. If I give timeAllowed a value say, 10s or above do you think the above
scenario of valid count displayed in numFound, but doc list empty can ever
occur still, as there is more time before cut-off to retrieve at least one
doc ?
Thanks!
Mark
Re: "timeAllowed" param with "numFound" having a count value but doc
list is empty
Posted by Mark Robinson <ma...@gmail.com>.
Thanks much Bram!
Best,
Mark
On Wed, Sep 16, 2020 at 3:59 AM Bram Van Dam <br...@intix.eu> wrote:
> There are a couple of open issues related to the timeAllowed parameter.
> For instance it currently doesn't work on conjunction with the
> cursorMark parameter [1]. And on Solr 7 it doesn't work at all [2].
>
> But other than that, when users have a lot of query flexibility, it's a
> pretty good idea to limit them somehow. You don't want your users to
> blow up your servers.
>
> [1] https://issues.apache.org/jira/browse/SOLR-14413
>
> [2] https://issues.apache.org/jira/browse/SOLR-9882
>
> - Bram
>
> On 16/09/2020 03:04, Mark Robinson wrote:
> > Thanks Dominique!
> > So is this parameter generally recommended or not. I wanted to try with a
> > value of 10s. We are not using it now.
> > My goal is to prevent a query from running more than 10s on the solr
> server
> > and choking it.
> >
> > What is the general recommendation.
> >
> > Thanks!
> > Mark
> >
> > On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <
> dominique.bejean@eolya.fr>
> > wrote:
> >
> >> Hi,
> >>
> >> 1. Yes, your analysis is correct
> >>
> >> 2. Yes, it can occurs too with very slow query.
> >>
> >> Regards
> >>
> >> Dominique
> >>
> >> Le mar. 15 sept. 2020 à 15:14, Mark Robinson <ma...@gmail.com>
> a
> >> écrit :
> >>
> >>> Hi,
> >>>
> >>> When in a sample query I used "timeAllowed" as low as 10mS, I got value
> >> for
> >>>
> >>> "numFound" as say 2000, but no docs were returned. But when I increased
> >> the
> >>>
> >>> value for timeAllowed to be in seconds, never got this scenario.
> >>>
> >>>
> >>>
> >>> I have 2 qns:-
> >>>
> >>> 1. Why does numFound have a value like say 2000 or even 6000 but no
> >>>
> >>> documents actually returned. During document collection is calculation
> of
> >>>
> >>> numFound done first and doc collection later?. Is doc list empty
> >> because,by
> >>>
> >>> the time doc collection started the timeAllowed cut off took effect?
> >>>
> >>>
> >>>
> >>> 2. If I give timeAllowed a value say, 10s or above do you think the
> above
> >>>
> >>> scenario of valid count displayed in numFound, but doc list empty can
> >> ever
> >>>
> >>> occur still, as there is more time before cut-off to retrieve at least
> >> one
> >>>
> >>> doc ?
> >>>
> >>>
> >>>
> >>> Thanks!
> >>>
> >>> Mark
> >>>
> >>>
> >>
> >
>
>
Re: "timeAllowed" param with "numFound" having a count value but doc
list is empty
Posted by Bram Van Dam <br...@intix.eu>.
There are a couple of open issues related to the timeAllowed parameter.
For instance it currently doesn't work on conjunction with the
cursorMark parameter [1]. And on Solr 7 it doesn't work at all [2].
But other than that, when users have a lot of query flexibility, it's a
pretty good idea to limit them somehow. You don't want your users to
blow up your servers.
[1] https://issues.apache.org/jira/browse/SOLR-14413
[2] https://issues.apache.org/jira/browse/SOLR-9882
- Bram
On 16/09/2020 03:04, Mark Robinson wrote:
> Thanks Dominique!
> So is this parameter generally recommended or not. I wanted to try with a
> value of 10s. We are not using it now.
> My goal is to prevent a query from running more than 10s on the solr server
> and choking it.
>
> What is the general recommendation.
>
> Thanks!
> Mark
>
> On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <do...@eolya.fr>
> wrote:
>
>> Hi,
>>
>> 1. Yes, your analysis is correct
>>
>> 2. Yes, it can occurs too with very slow query.
>>
>> Regards
>>
>> Dominique
>>
>> Le mar. 15 sept. 2020 à 15:14, Mark Robinson <ma...@gmail.com> a
>> écrit :
>>
>>> Hi,
>>>
>>> When in a sample query I used "timeAllowed" as low as 10mS, I got value
>> for
>>>
>>> "numFound" as say 2000, but no docs were returned. But when I increased
>> the
>>>
>>> value for timeAllowed to be in seconds, never got this scenario.
>>>
>>>
>>>
>>> I have 2 qns:-
>>>
>>> 1. Why does numFound have a value like say 2000 or even 6000 but no
>>>
>>> documents actually returned. During document collection is calculation of
>>>
>>> numFound done first and doc collection later?. Is doc list empty
>> because,by
>>>
>>> the time doc collection started the timeAllowed cut off took effect?
>>>
>>>
>>>
>>> 2. If I give timeAllowed a value say, 10s or above do you think the above
>>>
>>> scenario of valid count displayed in numFound, but doc list empty can
>> ever
>>>
>>> occur still, as there is more time before cut-off to retrieve at least
>> one
>>>
>>> doc ?
>>>
>>>
>>>
>>> Thanks!
>>>
>>> Mark
>>>
>>>
>>
>
Re: "timeAllowed" param with "numFound" having a count value but doc
list is empty
Posted by Mark Robinson <ma...@gmail.com>.
Thanks Colvin!
All the responses were helpful.
Best
Mark
On Wed, Sep 16, 2020 at 4:06 AM Colvin Cowie <co...@gmail.com>
wrote:
> Hi Mark,
>
> If queries taking 10 (or however many) seconds isn't acceptable, then
> either you need to a) prevent or optimize those queries, b) improve the
> performance of your index, c) use timeAllowed and accept that queries
> taking that long may fail or provide incomplete results, or d) a
> combination of the above.
>
> If you use timeAllowed then you have to accept the possibility that a query
> won't complete within the time allowed. Therefore you need to be able to
> deal with the possibility of the query failing or of it returning
> incomplete results.
>
> In our use of Solr, if a query exceeds timeAllowed we always treat it as a
> failure, even if it might have returned partial results, and return a 5xx
> response from our own server since we don't want to serve incomplete
> results ever. But you could attempt to return whatever results you do
> receive, perhaps with a warning message for your client indicating what
> happened.
>
>
> On Wed, 16 Sep 2020 at 02:05, Mark Robinson <ma...@gmail.com>
> wrote:
>
> > Thanks Dominique!
> > So is this parameter generally recommended or not. I wanted to try with a
> > value of 10s. We are not using it now.
> > My goal is to prevent a query from running more than 10s on the solr
> server
> > and choking it.
> >
> > What is the general recommendation.
> >
> > Thanks!
> > Mark
> >
> > On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <
> > dominique.bejean@eolya.fr>
> > wrote:
> >
> > > Hi,
> > >
> > > 1. Yes, your analysis is correct
> > >
> > > 2. Yes, it can occurs too with very slow query.
> > >
> > > Regards
> > >
> > > Dominique
> > >
> > > Le mar. 15 sept. 2020 à 15:14, Mark Robinson <ma...@gmail.com>
> a
> > > écrit :
> > >
> > > > Hi,
> > > >
> > > > When in a sample query I used "timeAllowed" as low as 10mS, I got
> value
> > > for
> > > >
> > > > "numFound" as say 2000, but no docs were returned. But when I
> increased
> > > the
> > > >
> > > > value for timeAllowed to be in seconds, never got this scenario.
> > > >
> > > >
> > > >
> > > > I have 2 qns:-
> > > >
> > > > 1. Why does numFound have a value like say 2000 or even 6000 but no
> > > >
> > > > documents actually returned. During document collection is
> calculation
> > of
> > > >
> > > > numFound done first and doc collection later?. Is doc list empty
> > > because,by
> > > >
> > > > the time doc collection started the timeAllowed cut off took effect?
> > > >
> > > >
> > > >
> > > > 2. If I give timeAllowed a value say, 10s or above do you think the
> > above
> > > >
> > > > scenario of valid count displayed in numFound, but doc list empty can
> > > ever
> > > >
> > > > occur still, as there is more time before cut-off to retrieve at
> least
> > > one
> > > >
> > > > doc ?
> > > >
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > Mark
> > > >
> > > >
> > >
> >
>
Re: "timeAllowed" param with "numFound" having a count value but doc
list is empty
Posted by Colvin Cowie <co...@gmail.com>.
Hi Mark,
If queries taking 10 (or however many) seconds isn't acceptable, then
either you need to a) prevent or optimize those queries, b) improve the
performance of your index, c) use timeAllowed and accept that queries
taking that long may fail or provide incomplete results, or d) a
combination of the above.
If you use timeAllowed then you have to accept the possibility that a query
won't complete within the time allowed. Therefore you need to be able to
deal with the possibility of the query failing or of it returning
incomplete results.
In our use of Solr, if a query exceeds timeAllowed we always treat it as a
failure, even if it might have returned partial results, and return a 5xx
response from our own server since we don't want to serve incomplete
results ever. But you could attempt to return whatever results you do
receive, perhaps with a warning message for your client indicating what
happened.
On Wed, 16 Sep 2020 at 02:05, Mark Robinson <ma...@gmail.com> wrote:
> Thanks Dominique!
> So is this parameter generally recommended or not. I wanted to try with a
> value of 10s. We are not using it now.
> My goal is to prevent a query from running more than 10s on the solr server
> and choking it.
>
> What is the general recommendation.
>
> Thanks!
> Mark
>
> On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <
> dominique.bejean@eolya.fr>
> wrote:
>
> > Hi,
> >
> > 1. Yes, your analysis is correct
> >
> > 2. Yes, it can occurs too with very slow query.
> >
> > Regards
> >
> > Dominique
> >
> > Le mar. 15 sept. 2020 à 15:14, Mark Robinson <ma...@gmail.com> a
> > écrit :
> >
> > > Hi,
> > >
> > > When in a sample query I used "timeAllowed" as low as 10mS, I got value
> > for
> > >
> > > "numFound" as say 2000, but no docs were returned. But when I increased
> > the
> > >
> > > value for timeAllowed to be in seconds, never got this scenario.
> > >
> > >
> > >
> > > I have 2 qns:-
> > >
> > > 1. Why does numFound have a value like say 2000 or even 6000 but no
> > >
> > > documents actually returned. During document collection is calculation
> of
> > >
> > > numFound done first and doc collection later?. Is doc list empty
> > because,by
> > >
> > > the time doc collection started the timeAllowed cut off took effect?
> > >
> > >
> > >
> > > 2. If I give timeAllowed a value say, 10s or above do you think the
> above
> > >
> > > scenario of valid count displayed in numFound, but doc list empty can
> > ever
> > >
> > > occur still, as there is more time before cut-off to retrieve at least
> > one
> > >
> > > doc ?
> > >
> > >
> > >
> > > Thanks!
> > >
> > > Mark
> > >
> > >
> >
>
Re: "timeAllowed" param with "numFound" having a count value but doc
list is empty
Posted by Mark Robinson <ma...@gmail.com>.
Thanks Dominique!
So is this parameter generally recommended or not. I wanted to try with a
value of 10s. We are not using it now.
My goal is to prevent a query from running more than 10s on the solr server
and choking it.
What is the general recommendation.
Thanks!
Mark
On Tue, Sep 15, 2020 at 5:38 PM Dominique Bejean <do...@eolya.fr>
wrote:
> Hi,
>
> 1. Yes, your analysis is correct
>
> 2. Yes, it can occurs too with very slow query.
>
> Regards
>
> Dominique
>
> Le mar. 15 sept. 2020 à 15:14, Mark Robinson <ma...@gmail.com> a
> écrit :
>
> > Hi,
> >
> > When in a sample query I used "timeAllowed" as low as 10mS, I got value
> for
> >
> > "numFound" as say 2000, but no docs were returned. But when I increased
> the
> >
> > value for timeAllowed to be in seconds, never got this scenario.
> >
> >
> >
> > I have 2 qns:-
> >
> > 1. Why does numFound have a value like say 2000 or even 6000 but no
> >
> > documents actually returned. During document collection is calculation of
> >
> > numFound done first and doc collection later?. Is doc list empty
> because,by
> >
> > the time doc collection started the timeAllowed cut off took effect?
> >
> >
> >
> > 2. If I give timeAllowed a value say, 10s or above do you think the above
> >
> > scenario of valid count displayed in numFound, but doc list empty can
> ever
> >
> > occur still, as there is more time before cut-off to retrieve at least
> one
> >
> > doc ?
> >
> >
> >
> > Thanks!
> >
> > Mark
> >
> >
>
Re: "timeAllowed" param with "numFound" having a count value but doc
list is empty
Posted by Dominique Bejean <do...@eolya.fr>.
Hi,
1. Yes, your analysis is correct
2. Yes, it can occurs too with very slow query.
Regards
Dominique
Le mar. 15 sept. 2020 à 15:14, Mark Robinson <ma...@gmail.com> a
écrit :
> Hi,
>
> When in a sample query I used "timeAllowed" as low as 10mS, I got value for
>
> "numFound" as say 2000, but no docs were returned. But when I increased the
>
> value for timeAllowed to be in seconds, never got this scenario.
>
>
>
> I have 2 qns:-
>
> 1. Why does numFound have a value like say 2000 or even 6000 but no
>
> documents actually returned. During document collection is calculation of
>
> numFound done first and doc collection later?. Is doc list empty because,by
>
> the time doc collection started the timeAllowed cut off took effect?
>
>
>
> 2. If I give timeAllowed a value say, 10s or above do you think the above
>
> scenario of valid count displayed in numFound, but doc list empty can ever
>
> occur still, as there is more time before cut-off to retrieve at least one
>
> doc ?
>
>
>
> Thanks!
>
> Mark
>
>