You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Stephen Lewis Bianamara <st...@gmail.com> on 2020/05/11 19:15:39 UTC
What is the logical order of applying sorts in SOLR?
Hi SOLR Community,
What is the order of operations which SOLR applies to sorting? I've
observed many times and across SOLR versions that a restrictive filter with
a sort takes an extremely long time to return, suggesting to me that the
SORT is applied before the filter.
An example situation is querying for fq:Foo=Bar vs querying for fq:Foo=Bar
sort by Id desc. I've observed over many SOLR versions and collections that
the former is orders of magnitude cheaper and quicker to respond, even when
the result set is tiny (10-100).
Does anyone in this forum know whether this is the default behavior and
whether there is any way through the API or SOLR configuration to apply
sorts after filters?
Thanks,
Stephen
Re: What is the logical order of applying sorts in SOLR?
Posted by Stephen Lewis Bianamara <st...@gmail.com>.
> If you use sort, you are basically ignoring relevancy
That's correct -- this is for querying with a stable sort, not for natural
language. One place this comes up for example is with cursors
<https://lucene.apache.org/solr/guide/8_0/pagination-of-results.html> --
cursors require the unique key be part of the sort.
> Do you see performance drop on non-clustered or clustered Solr? Because,
I would not be surprised if, for clustered node, all the results need to be
brought into one place to sort even if only 10 (of say 100) would be sent
back, where without sort, each node is asked for their "top X" matches and
others are never even sent
Good consideration here. The largest non clustered SOLR instance I have on
hand has on the order of 1M docs, but I'm also considering ones in the 1B
to 10B range. On this small one shard one I am not seeing the same loss of
performance, though it's so small in the first place that sorting 1M docs
should be snappy anyway.
It sounds like an interesting case to consider then would be to generate a
1B doc index on one node and two nodes, and see if the distinction is
isolated between these two cases. I would be happy to investigate that soon
and report back.
I think it would be good to return to the original question too -- is SOLR
applying the sort before applying filters? If I have an index with 1
billion docs, but a query which matches 100 docs, the sort should be
exceedingly cheap. Even if each shard is sorting up to 100 docs and sending
them back, if the filter is applied before the sort I would still expect
this to be extremely cheap.
On Wed, May 20, 2020 at 2:19 PM Alexandre Rafalovitch <ar...@gmail.com>
wrote:
> If you use sort, you are basically ignoring relevancy (unless you put
> that into sort). Which you seem to know as your example uses FQ.
>
> Do you see performance drop on non-clustered or clustered Solr?
> Because, I would not be surprised if, for clustered node, all the
> results need to be brought into one place to sort even if only 10 (of
> say 100) would be sent back, where without sort, each node is asked
> for their "top X" matches and others are never even sent. That would
> be my working theory anyway, I am not deep into milti-path mode the
> cluster code does.
>
> Regards,
> Alex.
>
> On Mon, 11 May 2020 at 15:16, Stephen Lewis Bianamara
> <st...@gmail.com> wrote:
> >
> > Hi SOLR Community,
> >
> > What is the order of operations which SOLR applies to sorting? I've
> > observed many times and across SOLR versions that a restrictive filter
> with
> > a sort takes an extremely long time to return, suggesting to me that the
> > SORT is applied before the filter.
> >
> > An example situation is querying for fq:Foo=Bar vs querying for
> fq:Foo=Bar
> > sort by Id desc. I've observed over many SOLR versions and collections
> that
> > the former is orders of magnitude cheaper and quicker to respond, even
> when
> > the result set is tiny (10-100).
> >
> > Does anyone in this forum know whether this is the default behavior and
> > whether there is any way through the API or SOLR configuration to apply
> > sorts after filters?
> >
> > Thanks,
> > Stephen
>
Re: What is the logical order of applying sorts in SOLR?
Posted by Alexandre Rafalovitch <ar...@gmail.com>.
If you use sort, you are basically ignoring relevancy (unless you put
that into sort). Which you seem to know as your example uses FQ.
Do you see performance drop on non-clustered or clustered Solr?
Because, I would not be surprised if, for clustered node, all the
results need to be brought into one place to sort even if only 10 (of
say 100) would be sent back, where without sort, each node is asked
for their "top X" matches and others are never even sent. That would
be my working theory anyway, I am not deep into milti-path mode the
cluster code does.
Regards,
Alex.
On Mon, 11 May 2020 at 15:16, Stephen Lewis Bianamara
<st...@gmail.com> wrote:
>
> Hi SOLR Community,
>
> What is the order of operations which SOLR applies to sorting? I've
> observed many times and across SOLR versions that a restrictive filter with
> a sort takes an extremely long time to return, suggesting to me that the
> SORT is applied before the filter.
>
> An example situation is querying for fq:Foo=Bar vs querying for fq:Foo=Bar
> sort by Id desc. I've observed over many SOLR versions and collections that
> the former is orders of magnitude cheaper and quicker to respond, even when
> the result set is tiny (10-100).
>
> Does anyone in this forum know whether this is the default behavior and
> whether there is any way through the API or SOLR configuration to apply
> sorts after filters?
>
> Thanks,
> Stephen
Re: What is the logical order of applying sorts in SOLR?
Posted by Stephen Lewis Bianamara <st...@gmail.com>.
I don't see how that would work. A boost would increase the relevance based
on a query, so how would it be able to apply a sort by the unique key field?
On Sat, May 16, 2020 at 3:51 PM David Hastings <ha...@gmail.com>
wrote:
> the bq parameter, heres a SO thread for it:
>
>
> https://stackoverflow.com/questions/45150856/how-to-know-when-to-use-solr-bq-vs-bf-and-how-to-apply-query-logic
>
> On Sat, May 16, 2020 at 6:27 PM Stephen Lewis Bianamara <
> stephen.bianamara@gmail.com> wrote:
>
> > Hi Paras,
> >
> > I'm not sure I follow. How would one replace sort by id with boosting?
> >
> > Thanks,
> > Stephen
> >
> > On Fri, May 15, 2020, 7:43 AM Paras Lehana
> > <pa...@indiamart.com.invalid> wrote:
> >
> > > As a workaround, can you try field boosting?
> > >
> > > On Tue, 12 May 2020 at 00:45, Stephen Lewis Bianamara <
> > > stephen.bianamara@gmail.com> wrote:
> > >
> > > > Hi SOLR Community,
> > > >
> > > > What is the order of operations which SOLR applies to sorting? I've
> > > > observed many times and across SOLR versions that a restrictive
> filter
> > > with
> > > > a sort takes an extremely long time to return, suggesting to me that
> > the
> > > > SORT is applied before the filter.
> > > >
> > > > An example situation is querying for fq:Foo=Bar vs querying for
> > > fq:Foo=Bar
> > > > sort by Id desc. I've observed over many SOLR versions and
> collections
> > > that
> > > > the former is orders of magnitude cheaper and quicker to respond,
> even
> > > when
> > > > the result set is tiny (10-100).
> > > >
> > > > Does anyone in this forum know whether this is the default behavior
> and
> > > > whether there is any way through the API or SOLR configuration to
> apply
> > > > sorts after filters?
> > > >
> > > > Thanks,
> > > > Stephen
> > > >
> > >
> > >
> > > --
> > > --
> > > Regards,
> > >
> > > *Paras Lehana* [65871]
> > > Development Engineer, *Auto-Suggest*,
> > > IndiaMART InterMESH Ltd,
> > >
> > > 11th Floor, Tower 2, Assotech Business Cresterra,
> > > Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305
> > >
> > > Mob.: +91-9560911996
> > > Work: 0120-4056700 | Extn:
> > > *1196*
> > >
> > > --
> > > *
> > > *
> > >
> > > <https://www.facebook.com/IndiaMART/videos/578196442936091/>
> > >
> >
>
Re: What is the logical order of applying sorts in SOLR?
Posted by David Hastings <ha...@gmail.com>.
the bq parameter, heres a SO thread for it:
https://stackoverflow.com/questions/45150856/how-to-know-when-to-use-solr-bq-vs-bf-and-how-to-apply-query-logic
On Sat, May 16, 2020 at 6:27 PM Stephen Lewis Bianamara <
stephen.bianamara@gmail.com> wrote:
> Hi Paras,
>
> I'm not sure I follow. How would one replace sort by id with boosting?
>
> Thanks,
> Stephen
>
> On Fri, May 15, 2020, 7:43 AM Paras Lehana
> <pa...@indiamart.com.invalid> wrote:
>
> > As a workaround, can you try field boosting?
> >
> > On Tue, 12 May 2020 at 00:45, Stephen Lewis Bianamara <
> > stephen.bianamara@gmail.com> wrote:
> >
> > > Hi SOLR Community,
> > >
> > > What is the order of operations which SOLR applies to sorting? I've
> > > observed many times and across SOLR versions that a restrictive filter
> > with
> > > a sort takes an extremely long time to return, suggesting to me that
> the
> > > SORT is applied before the filter.
> > >
> > > An example situation is querying for fq:Foo=Bar vs querying for
> > fq:Foo=Bar
> > > sort by Id desc. I've observed over many SOLR versions and collections
> > that
> > > the former is orders of magnitude cheaper and quicker to respond, even
> > when
> > > the result set is tiny (10-100).
> > >
> > > Does anyone in this forum know whether this is the default behavior and
> > > whether there is any way through the API or SOLR configuration to apply
> > > sorts after filters?
> > >
> > > Thanks,
> > > Stephen
> > >
> >
> >
> > --
> > --
> > Regards,
> >
> > *Paras Lehana* [65871]
> > Development Engineer, *Auto-Suggest*,
> > IndiaMART InterMESH Ltd,
> >
> > 11th Floor, Tower 2, Assotech Business Cresterra,
> > Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305
> >
> > Mob.: +91-9560911996
> > Work: 0120-4056700 | Extn:
> > *1196*
> >
> > --
> > *
> > *
> >
> > <https://www.facebook.com/IndiaMART/videos/578196442936091/>
> >
>
Re: What is the logical order of applying sorts in SOLR?
Posted by Stephen Lewis Bianamara <st...@gmail.com>.
Hi Paras,
I'm not sure I follow. How would one replace sort by id with boosting?
Thanks,
Stephen
On Fri, May 15, 2020, 7:43 AM Paras Lehana
<pa...@indiamart.com.invalid> wrote:
> As a workaround, can you try field boosting?
>
> On Tue, 12 May 2020 at 00:45, Stephen Lewis Bianamara <
> stephen.bianamara@gmail.com> wrote:
>
> > Hi SOLR Community,
> >
> > What is the order of operations which SOLR applies to sorting? I've
> > observed many times and across SOLR versions that a restrictive filter
> with
> > a sort takes an extremely long time to return, suggesting to me that the
> > SORT is applied before the filter.
> >
> > An example situation is querying for fq:Foo=Bar vs querying for
> fq:Foo=Bar
> > sort by Id desc. I've observed over many SOLR versions and collections
> that
> > the former is orders of magnitude cheaper and quicker to respond, even
> when
> > the result set is tiny (10-100).
> >
> > Does anyone in this forum know whether this is the default behavior and
> > whether there is any way through the API or SOLR configuration to apply
> > sorts after filters?
> >
> > Thanks,
> > Stephen
> >
>
>
> --
> --
> Regards,
>
> *Paras Lehana* [65871]
> Development Engineer, *Auto-Suggest*,
> IndiaMART InterMESH Ltd,
>
> 11th Floor, Tower 2, Assotech Business Cresterra,
> Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305
>
> Mob.: +91-9560911996
> Work: 0120-4056700 | Extn:
> *1196*
>
> --
> *
> *
>
> <https://www.facebook.com/IndiaMART/videos/578196442936091/>
>
Re: What is the logical order of applying sorts in SOLR?
Posted by Paras Lehana <pa...@indiamart.com.INVALID>.
As a workaround, can you try field boosting?
On Tue, 12 May 2020 at 00:45, Stephen Lewis Bianamara <
stephen.bianamara@gmail.com> wrote:
> Hi SOLR Community,
>
> What is the order of operations which SOLR applies to sorting? I've
> observed many times and across SOLR versions that a restrictive filter with
> a sort takes an extremely long time to return, suggesting to me that the
> SORT is applied before the filter.
>
> An example situation is querying for fq:Foo=Bar vs querying for fq:Foo=Bar
> sort by Id desc. I've observed over many SOLR versions and collections that
> the former is orders of magnitude cheaper and quicker to respond, even when
> the result set is tiny (10-100).
>
> Does anyone in this forum know whether this is the default behavior and
> whether there is any way through the API or SOLR configuration to apply
> sorts after filters?
>
> Thanks,
> Stephen
>
--
--
Regards,
*Paras Lehana* [65871]
Development Engineer, *Auto-Suggest*,
IndiaMART InterMESH Ltd,
11th Floor, Tower 2, Assotech Business Cresterra,
Plot No. 22, Sector 135, Noida, Uttar Pradesh, India 201305
Mob.: +91-9560911996
Work: 0120-4056700 | Extn:
*1196*
--
*
*
<https://www.facebook.com/IndiaMART/videos/578196442936091/>