You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Parshant Kumar <pa...@indiamart.com.INVALID> on 2021/03/08 04:43:26 UTC
Re: Increase in response time in case of collapse queries.
anyone please help
On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <pa...@indiamart.com>
wrote:
> Hi all,
>
> We have implemented collapse queries in place of grouped queries on our
> production solr. As mentioned in solr documentation collapse queries are
> recommended in place of grouped queries in terms of performance . But after
> switching to collapsed queries from grouped queries response time of
> queries have increased. This is unexpected behaviour, the response time
> should have been improved but results are opposites.
> Please someone help why response time is increased for collapsed queries.
>
> Thanks
> Parshant Kumar
>
--
Re: Increase in response time in case of collapse queries.
Posted by Gajendra Dadheech <ga...@gmail.com>.
Hey Florin. What was incremental benefit of this optimization of have group
field null for unique docs.
On Mon, Mar 8, 2021, 4:41 PM Florin Babes <ba...@gmail.com> wrote:
> We improved the performance of collapse by making the group_field null for
> the documents that have an unique value for group_field. This might help/
>
>
> În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> <pa...@indiamart.com.invalid> a scris:
>
> > yes,group_field is having high cardinality.
> >
> >
> > Thanks
> > Parshant Kumar
> >
> > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <ba...@gmail.com>
> wrote:
> >
> > > Your group_field has a high cardinality?
> > > Thanks,
> > > Florin Babes
> > >
> > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > <pa...@indiamart.com.invalid> a scris:
> > >
> > > > Hi florin,
> > > >
> > > > I am using below.
> > > >
> > > > 1) fq={!collapse field=parentglusrid}
> > > > 2) expand.rows=4
> > > > 3) expand=true
> > > >
> > > > Size of index is around 100GB.
> > > > Solr version is 6.5
> > > >
> > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <ba...@gmail.com>
> > > wrote:
> > > >
> > > > > Hello,
> > > > > First let's call the field you collapse on group_field
> > > > > If group_field has a high cardinality you should make group_field
> > null
> > > > for
> > > > > those documents that have a unique group_field and set
> > > nullPolicy=expand.
> > > > > By doing that solr will use less memory for it's internal maps (so
> > > faster
> > > > > gc) and the head selecting will be faster.
> > > > > What is your head selecting strategy? Can you share your fq which
> you
> > > use
> > > > > for collapsing?
> > > > >
> > > > > Thanks,
> > > > > Florin Babes
> > > > >
> > > > >
> > > > >
> > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > <pa...@indiamart.com.invalid> a scris:
> > > > >
> > > > > > anyone please help
> > > > > >
> > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > parshant.kumar@indiamart.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have implemented collapse queries in place of grouped
> queries
> > on
> > > > our
> > > > > > > production solr. As mentioned in solr documentation collapse
> > > queries
> > > > > are
> > > > > > > recommended in place of grouped queries in terms of
> performance .
> > > But
> > > > > > after
> > > > > > > switching to collapsed queries from grouped queries response
> time
> > > of
> > > > > > > queries have increased. This is unexpected behaviour, the
> > response
> > > > time
> > > > > > > should have been improved but results are opposites.
> > > > > > > Please someone help why response time is increased for
> collapsed
> > > > > queries.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Parshant Kumar
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>
Re: Increase in response time in case of collapse queries.
Posted by Joel Bernstein <jo...@gmail.com>.
So here are the response times:
Response time for the grouped method is 167 ms for 0.65 million requests.
Response time for the collapsed method is 177 ms for 0.65 million requests.
If group.ngroups is used with 6+ million cardinality, then the result set
size before grouping must have been small. With a large result set grouping
would have been quite slow in this scenario.
With small result sets, grouping is just fine also.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Mar 11, 2021 at 1:40 PM Gajendra Dadheech <ga...@gmail.com>
wrote:
> @florin
>
> Great advice. Null key for unique documents is really helpful. Any other
> such tricks that you are using to improve collapse performance ?
>
> On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar
> <pa...@indiamart.com.invalid> wrote:
>
> > Hi Joel,
> >
> > 1) What are the response times for both methods. Saying one is faster is
> > not specific enough.
> >
> > Response time for the grouped method is 167 ms for 0.65 million requests.
> > Response time for the collapsed method is 177 ms for 0.65 million
> requests.
> >
> > 2) What is the cardinality of the collapse field, saying it's high is not
> > specific enough. What is the actual cardinality?
> >
> > Cardinality of the collapse field is around 6.2 Million
> >
> > [image: image.png]
> > 3) Is ngroups used in the grouping query
> >
> > Yes, ngroups is used in grouping query.
> >
> > Thanks
> > Parshant Kumar
> >
> >
> >
> >
> > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <jo...@gmail.com>
> wrote:
> >
> >> Collapse is designed to outperform grouping in the following scenario:
> >>
> >> There is high cardinality on the group field and group.ngroups is
> needed.
> >> If either of these conditions is not satisfied grouping will typically
> be
> >> faster.
> >>
> >> You will need to provide some more information about your setup to get
> an
> >> answer to the collapse performance question.
> >>
> >> 1) What are the response times for both methods. Saying one is faster is
> >> not specific enough.
> >> 2) What is the cardinality of the collapse field, saying it's high is
> not
> >> specific enough. What is the actual cardinality?
> >> 3) Is ngroups used in the grouping query.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <ga...@gmail.com>
> >> wrote:
> >>
> >> > @prashant Florin means to put null for parentglusrid in documents
> where
> >> > this field-value is only present in one document [Group has only one
> >> > document]. and then use nullPolicy to include/expand.
> >> >
> >> >
> >> >
> >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> >> > <pa...@indiamart.com.invalid> wrote:
> >> >
> >> > > client should set to null the field if it's unique.
> >> > >
> >> > > @florin @Gajendra can you please explain more .I am not clear how to
> >> > > perform this.
> >> > >
> >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <ba...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > @Gajendra Our response time dropped by 36% and our rps increased
> >> with
> >> > > 27%.
> >> > > >
> >> > > > You have to reindex the core and the client should set to null the
> >> > field
> >> > > if
> >> > > > it's unique.
> >> > > >
> >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> >> > > > <pa...@indiamart.com.invalid> a scris:
> >> > > >
> >> > > > > How can we make group_field null? Using nullPolicy=expand ?
> >> > > > >
> >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <
> >> babesflorin@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > > We improved the performance of collapse by making the
> >> group_field
> >> > > null
> >> > > > > for
> >> > > > > > the documents that have an unique value for group_field. This
> >> might
> >> > > > help/
> >> > > > > >
> >> > > > > >
> >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> >> > > > > > <pa...@indiamart.com.invalid> a scris:
> >> > > > > >
> >> > > > > > > yes,group_field is having high cardinality.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Thanks
> >> > > > > > > Parshant Kumar
> >> > > > > > >
> >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> >> > babesflorin@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Your group_field has a high cardinality?
> >> > > > > > > > Thanks,
> >> > > > > > > > Florin Babes
> >> > > > > > > >
> >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> >> > > > > > > > <pa...@indiamart.com.invalid> a scris:
> >> > > > > > > >
> >> > > > > > > > > Hi florin,
> >> > > > > > > > >
> >> > > > > > > > > I am using below.
> >> > > > > > > > >
> >> > > > > > > > > 1) fq={!collapse field=parentglusrid}
> >> > > > > > > > > 2) expand.rows=4
> >> > > > > > > > > 3) expand=true
> >> > > > > > > > >
> >> > > > > > > > > Size of index is around 100GB.
> >> > > > > > > > > Solr version is 6.5
> >> > > > > > > > >
> >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> >> > > > babesflorin@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hello,
> >> > > > > > > > > > First let's call the field you collapse on group_field
> >> > > > > > > > > > If group_field has a high cardinality you should make
> >> > > > group_field
> >> > > > > > > null
> >> > > > > > > > > for
> >> > > > > > > > > > those documents that have a unique group_field and set
> >> > > > > > > > nullPolicy=expand.
> >> > > > > > > > > > By doing that solr will use less memory for it's
> >> internal
> >> > > maps
> >> > > > > (so
> >> > > > > > > > faster
> >> > > > > > > > > > gc) and the head selecting will be faster.
> >> > > > > > > > > > What is your head selecting strategy? Can you share
> >> your fq
> >> > > > which
> >> > > > > > you
> >> > > > > > > > use
> >> > > > > > > > > > for collapsing?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Florin Babes
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> >> > > > > > > > > > <pa...@indiamart.com.invalid> a scris:
> >> > > > > > > > > >
> >> > > > > > > > > > > anyone please help
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> >> > > > > > > > > > > parshant.kumar@indiamart.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > We have implemented collapse queries in place of
> >> > grouped
> >> > > > > > queries
> >> > > > > > > on
> >> > > > > > > > > our
> >> > > > > > > > > > > > production solr. As mentioned in solr
> documentation
> >> > > > collapse
> >> > > > > > > > queries
> >> > > > > > > > > > are
> >> > > > > > > > > > > > recommended in place of grouped queries in terms
> of
> >> > > > > > performance .
> >> > > > > > > > But
> >> > > > > > > > > > > after
> >> > > > > > > > > > > > switching to collapsed queries from grouped
> queries
> >> > > > response
> >> > > > > > time
> >> > > > > > > > of
> >> > > > > > > > > > > > queries have increased. This is unexpected
> >> behaviour,
> >> > the
> >> > > > > > > response
> >> > > > > > > > > time
> >> > > > > > > > > > > > should have been improved but results are
> opposites.
> >> > > > > > > > > > > > Please someone help why response time is increased
> >> for
> >> > > > > > collapsed
> >> > > > > > > > > > queries.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks
> >> > > > > > > > > > > > Parshant Kumar
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > > --
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Increase in response time in case of collapse queries.
Posted by David Smiley <ds...@apache.org>.
Oh yeah, glad you found that. It's not really a bug but something to
document. The collapse filter can be conceptually thought of as something
that occurs last, and thus "sees" documents in the order of your sort
parameter. Thus don't specify redundant sort options inside the collapse
because it's redundant work.
~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley
On Wed, Mar 17, 2021 at 11:19 AM Florin Babes <ba...@gmail.com> wrote:
> Yes. It seems that if you specify sort="score desc" the collapser
> recomputes the score of the documents as far as I could understand from the
> solr's code.
> Thanks,
>
> Florin Babeş
> PHP Developer
>
> babesflorin@gmail.com
>
>
> În mie., 17 mar. 2021 la 15:52, Joel Bernstein <jo...@gmail.com> a
> scris:
>
> > That is an interesting finding. The default behavior is quite fast
> compared
> > to the sort approach.
> >
> >
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> >
> > On Wed, Mar 17, 2021 at 3:42 AM Florin Babes <ba...@gmail.com>
> > wrote:
> >
> > > @Gajendra we used a sort="score desc" even if we only use the head of
> the
> > > group. By removing this sort and leaving the default behavior
> (selecting
> > > the head with the highest score) our response time dropped with 42% in
> > our
> > > synthetic load tests.
> > >
> > >
> > > În joi, 11 mar. 2021 la 20:40 Gajendra Dadheech <ga...@gmail.com>
> a
> > > scris:
> > >
> > > > @florin
> > > >
> > > > Great advice. Null key for unique documents is really helpful. Any
> > other
> > > > such tricks that you are using to improve collapse performance ?
> > > >
> > > > On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar
> > > > <pa...@indiamart.com.invalid> wrote:
> > > >
> > > > > Hi Joel,
> > > > >
> > > > > 1) What are the response times for both methods. Saying one is
> faster
> > > is
> > > > > not specific enough.
> > > > >
> > > > > Response time for the grouped method is 167 ms for 0.65 million
> > > requests.
> > > > > Response time for the collapsed method is 177 ms for 0.65 million
> > > > requests.
> > > > >
> > > > > 2) What is the cardinality of the collapse field, saying it's high
> is
> > > not
> > > > > specific enough. What is the actual cardinality?
> > > > >
> > > > > Cardinality of the collapse field is around 6.2 Million
> > > > >
> > > > > [image: image.png]
> > > > > 3) Is ngroups used in the grouping query
> > > > >
> > > > > Yes, ngroups is used in grouping query.
> > > > >
> > > > > Thanks
> > > > > Parshant Kumar
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <joelsolr@gmail.com
> >
> > > > wrote:
> > > > >
> > > > >> Collapse is designed to outperform grouping in the following
> > scenario:
> > > > >>
> > > > >> There is high cardinality on the group field and group.ngroups is
> > > > needed.
> > > > >> If either of these conditions is not satisfied grouping will
> > typically
> > > > be
> > > > >> faster.
> > > > >>
> > > > >> You will need to provide some more information about your setup to
> > get
> > > > an
> > > > >> answer to the collapse performance question.
> > > > >>
> > > > >> 1) What are the response times for both methods. Saying one is
> > faster
> > > is
> > > > >> not specific enough.
> > > > >> 2) What is the cardinality of the collapse field, saying it's high
> > is
> > > > not
> > > > >> specific enough. What is the actual cardinality?
> > > > >> 3) Is ngroups used in the grouping query.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> Joel Bernstein
> > > > >> http://joelsolr.blogspot.com/
> > > > >>
> > > > >>
> > > > >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <
> > > gajju3588@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > @prashant Florin means to put null for parentglusrid in
> documents
> > > > where
> > > > >> > this field-value is only present in one document [Group has only
> > one
> > > > >> > document]. and then use nullPolicy to include/expand.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> > > > >> > <pa...@indiamart.com.invalid> wrote:
> > > > >> >
> > > > >> > > client should set to null the field if it's unique.
> > > > >> > >
> > > > >> > > @florin @Gajendra can you please explain more .I am not clear
> > how
> > > to
> > > > >> > > perform this.
> > > > >> > >
> > > > >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <
> > > babesflorin@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > @Gajendra Our response time dropped by 36% and our rps
> > increased
> > > > >> with
> > > > >> > > 27%.
> > > > >> > > >
> > > > >> > > > You have to reindex the core and the client should set to
> null
> > > the
> > > > >> > field
> > > > >> > > if
> > > > >> > > > it's unique.
> > > > >> > > >
> > > > >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> > > > >> > > > <pa...@indiamart.com.invalid> a scris:
> > > > >> > > >
> > > > >> > > > > How can we make group_field null? Using nullPolicy=expand
> ?
> > > > >> > > > >
> > > > >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <
> > > > >> babesflorin@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > We improved the performance of collapse by making the
> > > > >> group_field
> > > > >> > > null
> > > > >> > > > > for
> > > > >> > > > > > the documents that have an unique value for group_field.
> > > This
> > > > >> might
> > > > >> > > > help/
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > > >> > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > >> > > > > >
> > > > >> > > > > > > yes,group_field is having high cardinality.
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks
> > > > >> > > > > > > Parshant Kumar
> > > > >> > > > > > >
> > > > >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> > > > >> > babesflorin@gmail.com
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Your group_field has a high cardinality?
> > > > >> > > > > > > > Thanks,
> > > > >> > > > > > > > Florin Babes
> > > > >> > > > > > > >
> > > > >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > >> > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi florin,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I am using below.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > 1) fq={!collapse field=parentglusrid}
> > > > >> > > > > > > > > 2) expand.rows=4
> > > > >> > > > > > > > > 3) expand=true
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Size of index is around 100GB.
> > > > >> > > > > > > > > Solr version is 6.5
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > > > >> > > > babesflorin@gmail.com
> > > > >> > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Hello,
> > > > >> > > > > > > > > > First let's call the field you collapse on
> > > group_field
> > > > >> > > > > > > > > > If group_field has a high cardinality you should
> > > make
> > > > >> > > > group_field
> > > > >> > > > > > > null
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > those documents that have a unique group_field
> and
> > > set
> > > > >> > > > > > > > nullPolicy=expand.
> > > > >> > > > > > > > > > By doing that solr will use less memory for it's
> > > > >> internal
> > > > >> > > maps
> > > > >> > > > > (so
> > > > >> > > > > > > > faster
> > > > >> > > > > > > > > > gc) and the head selecting will be faster.
> > > > >> > > > > > > > > > What is your head selecting strategy? Can you
> > share
> > > > >> your fq
> > > > >> > > > which
> > > > >> > > > > > you
> > > > >> > > > > > > > use
> > > > >> > > > > > > > > > for collapsing?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > Florin Babes
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > >> > > > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > anyone please help
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar
> <
> > > > >> > > > > > > > > > > parshant.kumar@indiamart.com>
> > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Hi all,
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > We have implemented collapse queries in
> place
> > of
> > > > >> > grouped
> > > > >> > > > > > queries
> > > > >> > > > > > > on
> > > > >> > > > > > > > > our
> > > > >> > > > > > > > > > > > production solr. As mentioned in solr
> > > > documentation
> > > > >> > > > collapse
> > > > >> > > > > > > > queries
> > > > >> > > > > > > > > > are
> > > > >> > > > > > > > > > > > recommended in place of grouped queries in
> > terms
> > > > of
> > > > >> > > > > > performance .
> > > > >> > > > > > > > But
> > > > >> > > > > > > > > > > after
> > > > >> > > > > > > > > > > > switching to collapsed queries from grouped
> > > > queries
> > > > >> > > > response
> > > > >> > > > > > time
> > > > >> > > > > > > > of
> > > > >> > > > > > > > > > > > queries have increased. This is unexpected
> > > > >> behaviour,
> > > > >> > the
> > > > >> > > > > > > response
> > > > >> > > > > > > > > time
> > > > >> > > > > > > > > > > > should have been improved but results are
> > > > opposites.
> > > > >> > > > > > > > > > > > Please someone help why response time is
> > > increased
> > > > >> for
> > > > >> > > > > > collapsed
> > > > >> > > > > > > > > > queries.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thanks
> > > > >> > > > > > > > > > > > Parshant Kumar
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > --
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > --
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > --
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > > > --
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > > --
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Increase in response time in case of collapse queries.
Posted by Florin Babes <ba...@gmail.com>.
Yes. It seems that if you specify sort="score desc" the collapser
recomputes the score of the documents as far as I could understand from the
solr's code.
Thanks,
Florin Babeş
PHP Developer
babesflorin@gmail.com
În mie., 17 mar. 2021 la 15:52, Joel Bernstein <jo...@gmail.com> a scris:
> That is an interesting finding. The default behavior is quite fast compared
> to the sort approach.
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Wed, Mar 17, 2021 at 3:42 AM Florin Babes <ba...@gmail.com>
> wrote:
>
> > @Gajendra we used a sort="score desc" even if we only use the head of the
> > group. By removing this sort and leaving the default behavior (selecting
> > the head with the highest score) our response time dropped with 42% in
> our
> > synthetic load tests.
> >
> >
> > În joi, 11 mar. 2021 la 20:40 Gajendra Dadheech <ga...@gmail.com> a
> > scris:
> >
> > > @florin
> > >
> > > Great advice. Null key for unique documents is really helpful. Any
> other
> > > such tricks that you are using to improve collapse performance ?
> > >
> > > On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar
> > > <pa...@indiamart.com.invalid> wrote:
> > >
> > > > Hi Joel,
> > > >
> > > > 1) What are the response times for both methods. Saying one is faster
> > is
> > > > not specific enough.
> > > >
> > > > Response time for the grouped method is 167 ms for 0.65 million
> > requests.
> > > > Response time for the collapsed method is 177 ms for 0.65 million
> > > requests.
> > > >
> > > > 2) What is the cardinality of the collapse field, saying it's high is
> > not
> > > > specific enough. What is the actual cardinality?
> > > >
> > > > Cardinality of the collapse field is around 6.2 Million
> > > >
> > > > [image: image.png]
> > > > 3) Is ngroups used in the grouping query
> > > >
> > > > Yes, ngroups is used in grouping query.
> > > >
> > > > Thanks
> > > > Parshant Kumar
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <jo...@gmail.com>
> > > wrote:
> > > >
> > > >> Collapse is designed to outperform grouping in the following
> scenario:
> > > >>
> > > >> There is high cardinality on the group field and group.ngroups is
> > > needed.
> > > >> If either of these conditions is not satisfied grouping will
> typically
> > > be
> > > >> faster.
> > > >>
> > > >> You will need to provide some more information about your setup to
> get
> > > an
> > > >> answer to the collapse performance question.
> > > >>
> > > >> 1) What are the response times for both methods. Saying one is
> faster
> > is
> > > >> not specific enough.
> > > >> 2) What is the cardinality of the collapse field, saying it's high
> is
> > > not
> > > >> specific enough. What is the actual cardinality?
> > > >> 3) Is ngroups used in the grouping query.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> Joel Bernstein
> > > >> http://joelsolr.blogspot.com/
> > > >>
> > > >>
> > > >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <
> > gajju3588@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > @prashant Florin means to put null for parentglusrid in documents
> > > where
> > > >> > this field-value is only present in one document [Group has only
> one
> > > >> > document]. and then use nullPolicy to include/expand.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> > > >> > <pa...@indiamart.com.invalid> wrote:
> > > >> >
> > > >> > > client should set to null the field if it's unique.
> > > >> > >
> > > >> > > @florin @Gajendra can you please explain more .I am not clear
> how
> > to
> > > >> > > perform this.
> > > >> > >
> > > >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <
> > babesflorin@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > > @Gajendra Our response time dropped by 36% and our rps
> increased
> > > >> with
> > > >> > > 27%.
> > > >> > > >
> > > >> > > > You have to reindex the core and the client should set to null
> > the
> > > >> > field
> > > >> > > if
> > > >> > > > it's unique.
> > > >> > > >
> > > >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> > > >> > > > <pa...@indiamart.com.invalid> a scris:
> > > >> > > >
> > > >> > > > > How can we make group_field null? Using nullPolicy=expand ?
> > > >> > > > >
> > > >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <
> > > >> babesflorin@gmail.com>
> > > >> > > > wrote:
> > > >> > > > >
> > > >> > > > > > We improved the performance of collapse by making the
> > > >> group_field
> > > >> > > null
> > > >> > > > > for
> > > >> > > > > > the documents that have an unique value for group_field.
> > This
> > > >> might
> > > >> > > > help/
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > >> > > > > > <pa...@indiamart.com.invalid> a scris:
> > > >> > > > > >
> > > >> > > > > > > yes,group_field is having high cardinality.
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > Thanks
> > > >> > > > > > > Parshant Kumar
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> > > >> > babesflorin@gmail.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Your group_field has a high cardinality?
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > > Florin Babes
> > > >> > > > > > > >
> > > >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > >> > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi florin,
> > > >> > > > > > > > >
> > > >> > > > > > > > > I am using below.
> > > >> > > > > > > > >
> > > >> > > > > > > > > 1) fq={!collapse field=parentglusrid}
> > > >> > > > > > > > > 2) expand.rows=4
> > > >> > > > > > > > > 3) expand=true
> > > >> > > > > > > > >
> > > >> > > > > > > > > Size of index is around 100GB.
> > > >> > > > > > > > > Solr version is 6.5
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > > >> > > > babesflorin@gmail.com
> > > >> > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hello,
> > > >> > > > > > > > > > First let's call the field you collapse on
> > group_field
> > > >> > > > > > > > > > If group_field has a high cardinality you should
> > make
> > > >> > > > group_field
> > > >> > > > > > > null
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > those documents that have a unique group_field and
> > set
> > > >> > > > > > > > nullPolicy=expand.
> > > >> > > > > > > > > > By doing that solr will use less memory for it's
> > > >> internal
> > > >> > > maps
> > > >> > > > > (so
> > > >> > > > > > > > faster
> > > >> > > > > > > > > > gc) and the head selecting will be faster.
> > > >> > > > > > > > > > What is your head selecting strategy? Can you
> share
> > > >> your fq
> > > >> > > > which
> > > >> > > > > > you
> > > >> > > > > > > > use
> > > >> > > > > > > > > > for collapsing?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks,
> > > >> > > > > > > > > > Florin Babes
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > >> > > > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > anyone please help
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > >> > > > > > > > > > > parshant.kumar@indiamart.com>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Hi all,
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > We have implemented collapse queries in place
> of
> > > >> > grouped
> > > >> > > > > > queries
> > > >> > > > > > > on
> > > >> > > > > > > > > our
> > > >> > > > > > > > > > > > production solr. As mentioned in solr
> > > documentation
> > > >> > > > collapse
> > > >> > > > > > > > queries
> > > >> > > > > > > > > > are
> > > >> > > > > > > > > > > > recommended in place of grouped queries in
> terms
> > > of
> > > >> > > > > > performance .
> > > >> > > > > > > > But
> > > >> > > > > > > > > > > after
> > > >> > > > > > > > > > > > switching to collapsed queries from grouped
> > > queries
> > > >> > > > response
> > > >> > > > > > time
> > > >> > > > > > > > of
> > > >> > > > > > > > > > > > queries have increased. This is unexpected
> > > >> behaviour,
> > > >> > the
> > > >> > > > > > > response
> > > >> > > > > > > > > time
> > > >> > > > > > > > > > > > should have been improved but results are
> > > opposites.
> > > >> > > > > > > > > > > > Please someone help why response time is
> > increased
> > > >> for
> > > >> > > > > > collapsed
> > > >> > > > > > > > > > queries.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks
> > > >> > > > > > > > > > > > Parshant Kumar
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > --
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > --
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> > > --
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Increase in response time in case of collapse queries.
Posted by Joel Bernstein <jo...@gmail.com>.
That is an interesting finding. The default behavior is quite fast compared
to the sort approach.
Joel Bernstein
http://joelsolr.blogspot.com/
On Wed, Mar 17, 2021 at 3:42 AM Florin Babes <ba...@gmail.com> wrote:
> @Gajendra we used a sort="score desc" even if we only use the head of the
> group. By removing this sort and leaving the default behavior (selecting
> the head with the highest score) our response time dropped with 42% in our
> synthetic load tests.
>
>
> În joi, 11 mar. 2021 la 20:40 Gajendra Dadheech <ga...@gmail.com> a
> scris:
>
> > @florin
> >
> > Great advice. Null key for unique documents is really helpful. Any other
> > such tricks that you are using to improve collapse performance ?
> >
> > On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar
> > <pa...@indiamart.com.invalid> wrote:
> >
> > > Hi Joel,
> > >
> > > 1) What are the response times for both methods. Saying one is faster
> is
> > > not specific enough.
> > >
> > > Response time for the grouped method is 167 ms for 0.65 million
> requests.
> > > Response time for the collapsed method is 177 ms for 0.65 million
> > requests.
> > >
> > > 2) What is the cardinality of the collapse field, saying it's high is
> not
> > > specific enough. What is the actual cardinality?
> > >
> > > Cardinality of the collapse field is around 6.2 Million
> > >
> > > [image: image.png]
> > > 3) Is ngroups used in the grouping query
> > >
> > > Yes, ngroups is used in grouping query.
> > >
> > > Thanks
> > > Parshant Kumar
> > >
> > >
> > >
> > >
> > > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <jo...@gmail.com>
> > wrote:
> > >
> > >> Collapse is designed to outperform grouping in the following scenario:
> > >>
> > >> There is high cardinality on the group field and group.ngroups is
> > needed.
> > >> If either of these conditions is not satisfied grouping will typically
> > be
> > >> faster.
> > >>
> > >> You will need to provide some more information about your setup to get
> > an
> > >> answer to the collapse performance question.
> > >>
> > >> 1) What are the response times for both methods. Saying one is faster
> is
> > >> not specific enough.
> > >> 2) What is the cardinality of the collapse field, saying it's high is
> > not
> > >> specific enough. What is the actual cardinality?
> > >> 3) Is ngroups used in the grouping query.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Joel Bernstein
> > >> http://joelsolr.blogspot.com/
> > >>
> > >>
> > >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <
> gajju3588@gmail.com>
> > >> wrote:
> > >>
> > >> > @prashant Florin means to put null for parentglusrid in documents
> > where
> > >> > this field-value is only present in one document [Group has only one
> > >> > document]. and then use nullPolicy to include/expand.
> > >> >
> > >> >
> > >> >
> > >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> > >> > <pa...@indiamart.com.invalid> wrote:
> > >> >
> > >> > > client should set to null the field if it's unique.
> > >> > >
> > >> > > @florin @Gajendra can you please explain more .I am not clear how
> to
> > >> > > perform this.
> > >> > >
> > >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <
> babesflorin@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > > @Gajendra Our response time dropped by 36% and our rps increased
> > >> with
> > >> > > 27%.
> > >> > > >
> > >> > > > You have to reindex the core and the client should set to null
> the
> > >> > field
> > >> > > if
> > >> > > > it's unique.
> > >> > > >
> > >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> > >> > > > <pa...@indiamart.com.invalid> a scris:
> > >> > > >
> > >> > > > > How can we make group_field null? Using nullPolicy=expand ?
> > >> > > > >
> > >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <
> > >> babesflorin@gmail.com>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > We improved the performance of collapse by making the
> > >> group_field
> > >> > > null
> > >> > > > > for
> > >> > > > > > the documents that have an unique value for group_field.
> This
> > >> might
> > >> > > > help/
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > >> > > > > > <pa...@indiamart.com.invalid> a scris:
> > >> > > > > >
> > >> > > > > > > yes,group_field is having high cardinality.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > Thanks
> > >> > > > > > > Parshant Kumar
> > >> > > > > > >
> > >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> > >> > babesflorin@gmail.com
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Your group_field has a high cardinality?
> > >> > > > > > > > Thanks,
> > >> > > > > > > > Florin Babes
> > >> > > > > > > >
> > >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > >> > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > >> > > > > > > >
> > >> > > > > > > > > Hi florin,
> > >> > > > > > > > >
> > >> > > > > > > > > I am using below.
> > >> > > > > > > > >
> > >> > > > > > > > > 1) fq={!collapse field=parentglusrid}
> > >> > > > > > > > > 2) expand.rows=4
> > >> > > > > > > > > 3) expand=true
> > >> > > > > > > > >
> > >> > > > > > > > > Size of index is around 100GB.
> > >> > > > > > > > > Solr version is 6.5
> > >> > > > > > > > >
> > >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > >> > > > babesflorin@gmail.com
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hello,
> > >> > > > > > > > > > First let's call the field you collapse on
> group_field
> > >> > > > > > > > > > If group_field has a high cardinality you should
> make
> > >> > > > group_field
> > >> > > > > > > null
> > >> > > > > > > > > for
> > >> > > > > > > > > > those documents that have a unique group_field and
> set
> > >> > > > > > > > nullPolicy=expand.
> > >> > > > > > > > > > By doing that solr will use less memory for it's
> > >> internal
> > >> > > maps
> > >> > > > > (so
> > >> > > > > > > > faster
> > >> > > > > > > > > > gc) and the head selecting will be faster.
> > >> > > > > > > > > > What is your head selecting strategy? Can you share
> > >> your fq
> > >> > > > which
> > >> > > > > > you
> > >> > > > > > > > use
> > >> > > > > > > > > > for collapsing?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks,
> > >> > > > > > > > > > Florin Babes
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > >> > > > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > anyone please help
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > >> > > > > > > > > > > parshant.kumar@indiamart.com>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Hi all,
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > We have implemented collapse queries in place of
> > >> > grouped
> > >> > > > > > queries
> > >> > > > > > > on
> > >> > > > > > > > > our
> > >> > > > > > > > > > > > production solr. As mentioned in solr
> > documentation
> > >> > > > collapse
> > >> > > > > > > > queries
> > >> > > > > > > > > > are
> > >> > > > > > > > > > > > recommended in place of grouped queries in terms
> > of
> > >> > > > > > performance .
> > >> > > > > > > > But
> > >> > > > > > > > > > > after
> > >> > > > > > > > > > > > switching to collapsed queries from grouped
> > queries
> > >> > > > response
> > >> > > > > > time
> > >> > > > > > > > of
> > >> > > > > > > > > > > > queries have increased. This is unexpected
> > >> behaviour,
> > >> > the
> > >> > > > > > > response
> > >> > > > > > > > > time
> > >> > > > > > > > > > > > should have been improved but results are
> > opposites.
> > >> > > > > > > > > > > > Please someone help why response time is
> increased
> > >> for
> > >> > > > > > collapsed
> > >> > > > > > > > > > queries.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks
> > >> > > > > > > > > > > > Parshant Kumar
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> > > --
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Increase in response time in case of collapse queries.
Posted by Florin Babes <ba...@gmail.com>.
@Gajendra we used a sort="score desc" even if we only use the head of the
group. By removing this sort and leaving the default behavior (selecting
the head with the highest score) our response time dropped with 42% in our
synthetic load tests.
În joi, 11 mar. 2021 la 20:40 Gajendra Dadheech <ga...@gmail.com> a
scris:
> @florin
>
> Great advice. Null key for unique documents is really helpful. Any other
> such tricks that you are using to improve collapse performance ?
>
> On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar
> <pa...@indiamart.com.invalid> wrote:
>
> > Hi Joel,
> >
> > 1) What are the response times for both methods. Saying one is faster is
> > not specific enough.
> >
> > Response time for the grouped method is 167 ms for 0.65 million requests.
> > Response time for the collapsed method is 177 ms for 0.65 million
> requests.
> >
> > 2) What is the cardinality of the collapse field, saying it's high is not
> > specific enough. What is the actual cardinality?
> >
> > Cardinality of the collapse field is around 6.2 Million
> >
> > [image: image.png]
> > 3) Is ngroups used in the grouping query
> >
> > Yes, ngroups is used in grouping query.
> >
> > Thanks
> > Parshant Kumar
> >
> >
> >
> >
> > On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <jo...@gmail.com>
> wrote:
> >
> >> Collapse is designed to outperform grouping in the following scenario:
> >>
> >> There is high cardinality on the group field and group.ngroups is
> needed.
> >> If either of these conditions is not satisfied grouping will typically
> be
> >> faster.
> >>
> >> You will need to provide some more information about your setup to get
> an
> >> answer to the collapse performance question.
> >>
> >> 1) What are the response times for both methods. Saying one is faster is
> >> not specific enough.
> >> 2) What is the cardinality of the collapse field, saying it's high is
> not
> >> specific enough. What is the actual cardinality?
> >> 3) Is ngroups used in the grouping query.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >>
> >> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <ga...@gmail.com>
> >> wrote:
> >>
> >> > @prashant Florin means to put null for parentglusrid in documents
> where
> >> > this field-value is only present in one document [Group has only one
> >> > document]. and then use nullPolicy to include/expand.
> >> >
> >> >
> >> >
> >> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> >> > <pa...@indiamart.com.invalid> wrote:
> >> >
> >> > > client should set to null the field if it's unique.
> >> > >
> >> > > @florin @Gajendra can you please explain more .I am not clear how to
> >> > > perform this.
> >> > >
> >> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <ba...@gmail.com>
> >> > wrote:
> >> > >
> >> > > > @Gajendra Our response time dropped by 36% and our rps increased
> >> with
> >> > > 27%.
> >> > > >
> >> > > > You have to reindex the core and the client should set to null the
> >> > field
> >> > > if
> >> > > > it's unique.
> >> > > >
> >> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> >> > > > <pa...@indiamart.com.invalid> a scris:
> >> > > >
> >> > > > > How can we make group_field null? Using nullPolicy=expand ?
> >> > > > >
> >> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <
> >> babesflorin@gmail.com>
> >> > > > wrote:
> >> > > > >
> >> > > > > > We improved the performance of collapse by making the
> >> group_field
> >> > > null
> >> > > > > for
> >> > > > > > the documents that have an unique value for group_field. This
> >> might
> >> > > > help/
> >> > > > > >
> >> > > > > >
> >> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> >> > > > > > <pa...@indiamart.com.invalid> a scris:
> >> > > > > >
> >> > > > > > > yes,group_field is having high cardinality.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Thanks
> >> > > > > > > Parshant Kumar
> >> > > > > > >
> >> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> >> > babesflorin@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Your group_field has a high cardinality?
> >> > > > > > > > Thanks,
> >> > > > > > > > Florin Babes
> >> > > > > > > >
> >> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> >> > > > > > > > <pa...@indiamart.com.invalid> a scris:
> >> > > > > > > >
> >> > > > > > > > > Hi florin,
> >> > > > > > > > >
> >> > > > > > > > > I am using below.
> >> > > > > > > > >
> >> > > > > > > > > 1) fq={!collapse field=parentglusrid}
> >> > > > > > > > > 2) expand.rows=4
> >> > > > > > > > > 3) expand=true
> >> > > > > > > > >
> >> > > > > > > > > Size of index is around 100GB.
> >> > > > > > > > > Solr version is 6.5
> >> > > > > > > > >
> >> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> >> > > > babesflorin@gmail.com
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hello,
> >> > > > > > > > > > First let's call the field you collapse on group_field
> >> > > > > > > > > > If group_field has a high cardinality you should make
> >> > > > group_field
> >> > > > > > > null
> >> > > > > > > > > for
> >> > > > > > > > > > those documents that have a unique group_field and set
> >> > > > > > > > nullPolicy=expand.
> >> > > > > > > > > > By doing that solr will use less memory for it's
> >> internal
> >> > > maps
> >> > > > > (so
> >> > > > > > > > faster
> >> > > > > > > > > > gc) and the head selecting will be faster.
> >> > > > > > > > > > What is your head selecting strategy? Can you share
> >> your fq
> >> > > > which
> >> > > > > > you
> >> > > > > > > > use
> >> > > > > > > > > > for collapsing?
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks,
> >> > > > > > > > > > Florin Babes
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> >> > > > > > > > > > <pa...@indiamart.com.invalid> a scris:
> >> > > > > > > > > >
> >> > > > > > > > > > > anyone please help
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> >> > > > > > > > > > > parshant.kumar@indiamart.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > We have implemented collapse queries in place of
> >> > grouped
> >> > > > > > queries
> >> > > > > > > on
> >> > > > > > > > > our
> >> > > > > > > > > > > > production solr. As mentioned in solr
> documentation
> >> > > > collapse
> >> > > > > > > > queries
> >> > > > > > > > > > are
> >> > > > > > > > > > > > recommended in place of grouped queries in terms
> of
> >> > > > > > performance .
> >> > > > > > > > But
> >> > > > > > > > > > > after
> >> > > > > > > > > > > > switching to collapsed queries from grouped
> queries
> >> > > > response
> >> > > > > > time
> >> > > > > > > > of
> >> > > > > > > > > > > > queries have increased. This is unexpected
> >> behaviour,
> >> > the
> >> > > > > > > response
> >> > > > > > > > > time
> >> > > > > > > > > > > > should have been improved but results are
> opposites.
> >> > > > > > > > > > > > Please someone help why response time is increased
> >> for
> >> > > > > > collapsed
> >> > > > > > > > > > queries.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks
> >> > > > > > > > > > > > Parshant Kumar
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > > --
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Increase in response time in case of collapse queries.
Posted by Gajendra Dadheech <ga...@gmail.com>.
@florin
Great advice. Null key for unique documents is really helpful. Any other
such tricks that you are using to improve collapse performance ?
On Tue, Mar 9, 2021, 2:45 PM Parshant Kumar
<pa...@indiamart.com.invalid> wrote:
> Hi Joel,
>
> 1) What are the response times for both methods. Saying one is faster is
> not specific enough.
>
> Response time for the grouped method is 167 ms for 0.65 million requests.
> Response time for the collapsed method is 177 ms for 0.65 million requests.
>
> 2) What is the cardinality of the collapse field, saying it's high is not
> specific enough. What is the actual cardinality?
>
> Cardinality of the collapse field is around 6.2 Million
>
> [image: image.png]
> 3) Is ngroups used in the grouping query
>
> Yes, ngroups is used in grouping query.
>
> Thanks
> Parshant Kumar
>
>
>
>
> On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <jo...@gmail.com> wrote:
>
>> Collapse is designed to outperform grouping in the following scenario:
>>
>> There is high cardinality on the group field and group.ngroups is needed.
>> If either of these conditions is not satisfied grouping will typically be
>> faster.
>>
>> You will need to provide some more information about your setup to get an
>> answer to the collapse performance question.
>>
>> 1) What are the response times for both methods. Saying one is faster is
>> not specific enough.
>> 2) What is the cardinality of the collapse field, saying it's high is not
>> specific enough. What is the actual cardinality?
>> 3) Is ngroups used in the grouping query.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <ga...@gmail.com>
>> wrote:
>>
>> > @prashant Florin means to put null for parentglusrid in documents where
>> > this field-value is only present in one document [Group has only one
>> > document]. and then use nullPolicy to include/expand.
>> >
>> >
>> >
>> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
>> > <pa...@indiamart.com.invalid> wrote:
>> >
>> > > client should set to null the field if it's unique.
>> > >
>> > > @florin @Gajendra can you please explain more .I am not clear how to
>> > > perform this.
>> > >
>> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <ba...@gmail.com>
>> > wrote:
>> > >
>> > > > @Gajendra Our response time dropped by 36% and our rps increased
>> with
>> > > 27%.
>> > > >
>> > > > You have to reindex the core and the client should set to null the
>> > field
>> > > if
>> > > > it's unique.
>> > > >
>> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
>> > > > <pa...@indiamart.com.invalid> a scris:
>> > > >
>> > > > > How can we make group_field null? Using nullPolicy=expand ?
>> > > > >
>> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <
>> babesflorin@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > We improved the performance of collapse by making the
>> group_field
>> > > null
>> > > > > for
>> > > > > > the documents that have an unique value for group_field. This
>> might
>> > > > help/
>> > > > > >
>> > > > > >
>> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
>> > > > > > <pa...@indiamart.com.invalid> a scris:
>> > > > > >
>> > > > > > > yes,group_field is having high cardinality.
>> > > > > > >
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > > Parshant Kumar
>> > > > > > >
>> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
>> > babesflorin@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Your group_field has a high cardinality?
>> > > > > > > > Thanks,
>> > > > > > > > Florin Babes
>> > > > > > > >
>> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
>> > > > > > > > <pa...@indiamart.com.invalid> a scris:
>> > > > > > > >
>> > > > > > > > > Hi florin,
>> > > > > > > > >
>> > > > > > > > > I am using below.
>> > > > > > > > >
>> > > > > > > > > 1) fq={!collapse field=parentglusrid}
>> > > > > > > > > 2) expand.rows=4
>> > > > > > > > > 3) expand=true
>> > > > > > > > >
>> > > > > > > > > Size of index is around 100GB.
>> > > > > > > > > Solr version is 6.5
>> > > > > > > > >
>> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
>> > > > babesflorin@gmail.com
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hello,
>> > > > > > > > > > First let's call the field you collapse on group_field
>> > > > > > > > > > If group_field has a high cardinality you should make
>> > > > group_field
>> > > > > > > null
>> > > > > > > > > for
>> > > > > > > > > > those documents that have a unique group_field and set
>> > > > > > > > nullPolicy=expand.
>> > > > > > > > > > By doing that solr will use less memory for it's
>> internal
>> > > maps
>> > > > > (so
>> > > > > > > > faster
>> > > > > > > > > > gc) and the head selecting will be faster.
>> > > > > > > > > > What is your head selecting strategy? Can you share
>> your fq
>> > > > which
>> > > > > > you
>> > > > > > > > use
>> > > > > > > > > > for collapsing?
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Florin Babes
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
>> > > > > > > > > > <pa...@indiamart.com.invalid> a scris:
>> > > > > > > > > >
>> > > > > > > > > > > anyone please help
>> > > > > > > > > > >
>> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
>> > > > > > > > > > > parshant.kumar@indiamart.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi all,
>> > > > > > > > > > > >
>> > > > > > > > > > > > We have implemented collapse queries in place of
>> > grouped
>> > > > > > queries
>> > > > > > > on
>> > > > > > > > > our
>> > > > > > > > > > > > production solr. As mentioned in solr documentation
>> > > > collapse
>> > > > > > > > queries
>> > > > > > > > > > are
>> > > > > > > > > > > > recommended in place of grouped queries in terms of
>> > > > > > performance .
>> > > > > > > > But
>> > > > > > > > > > > after
>> > > > > > > > > > > > switching to collapsed queries from grouped queries
>> > > > response
>> > > > > > time
>> > > > > > > > of
>> > > > > > > > > > > > queries have increased. This is unexpected
>> behaviour,
>> > the
>> > > > > > > response
>> > > > > > > > > time
>> > > > > > > > > > > > should have been improved but results are opposites.
>> > > > > > > > > > > > Please someone help why response time is increased
>> for
>> > > > > > collapsed
>> > > > > > > > > > queries.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks
>> > > > > > > > > > > > Parshant Kumar
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > >
>> > > >
>> > >
>> > > --
>> > >
>> > >
>> >
>>
>
>
Re: Increase in response time in case of collapse queries.
Posted by Parshant Kumar <pa...@indiamart.com.INVALID>.
Hi Joel,
1) What are the response times for both methods. Saying one is faster is
not specific enough.
Response time for the grouped method is 167 ms for 0.65 million requests.
Response time for the collapsed method is 177 ms for 0.65 million requests.
2) What is the cardinality of the collapse field, saying it's high is not
specific enough. What is the actual cardinality?
Cardinality of the collapse field is around 6.2 Million
[image: image.png]
3) Is ngroups used in the grouping query
Yes, ngroups is used in grouping query.
Thanks
Parshant Kumar
On Tue, Mar 9, 2021 at 12:30 AM Joel Bernstein <jo...@gmail.com> wrote:
> Collapse is designed to outperform grouping in the following scenario:
>
> There is high cardinality on the group field and group.ngroups is needed.
> If either of these conditions is not satisfied grouping will typically be
> faster.
>
> You will need to provide some more information about your setup to get an
> answer to the collapse performance question.
>
> 1) What are the response times for both methods. Saying one is faster is
> not specific enough.
> 2) What is the cardinality of the collapse field, saying it's high is not
> specific enough. What is the actual cardinality?
> 3) Is ngroups used in the grouping query.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <ga...@gmail.com>
> wrote:
>
> > @prashant Florin means to put null for parentglusrid in documents where
> > this field-value is only present in one document [Group has only one
> > document]. and then use nullPolicy to include/expand.
> >
> >
> >
> > On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> > <pa...@indiamart.com.invalid> wrote:
> >
> > > client should set to null the field if it's unique.
> > >
> > > @florin @Gajendra can you please explain more .I am not clear how to
> > > perform this.
> > >
> > > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <ba...@gmail.com>
> > wrote:
> > >
> > > > @Gajendra Our response time dropped by 36% and our rps increased with
> > > 27%.
> > > >
> > > > You have to reindex the core and the client should set to null the
> > field
> > > if
> > > > it's unique.
> > > >
> > > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> > > > <pa...@indiamart.com.invalid> a scris:
> > > >
> > > > > How can we make group_field null? Using nullPolicy=expand ?
> > > > >
> > > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <babesflorin@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > We improved the performance of collapse by making the group_field
> > > null
> > > > > for
> > > > > > the documents that have an unique value for group_field. This
> might
> > > > help/
> > > > > >
> > > > > >
> > > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > >
> > > > > > > yes,group_field is having high cardinality.
> > > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > > Parshant Kumar
> > > > > > >
> > > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> > babesflorin@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Your group_field has a high cardinality?
> > > > > > > > Thanks,
> > > > > > > > Florin Babes
> > > > > > > >
> > > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > > > >
> > > > > > > > > Hi florin,
> > > > > > > > >
> > > > > > > > > I am using below.
> > > > > > > > >
> > > > > > > > > 1) fq={!collapse field=parentglusrid}
> > > > > > > > > 2) expand.rows=4
> > > > > > > > > 3) expand=true
> > > > > > > > >
> > > > > > > > > Size of index is around 100GB.
> > > > > > > > > Solr version is 6.5
> > > > > > > > >
> > > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > > > babesflorin@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > > First let's call the field you collapse on group_field
> > > > > > > > > > If group_field has a high cardinality you should make
> > > > group_field
> > > > > > > null
> > > > > > > > > for
> > > > > > > > > > those documents that have a unique group_field and set
> > > > > > > > nullPolicy=expand.
> > > > > > > > > > By doing that solr will use less memory for it's internal
> > > maps
> > > > > (so
> > > > > > > > faster
> > > > > > > > > > gc) and the head selecting will be faster.
> > > > > > > > > > What is your head selecting strategy? Can you share your
> fq
> > > > which
> > > > > > you
> > > > > > > > use
> > > > > > > > > > for collapsing?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Florin Babes
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > > > > > >
> > > > > > > > > > > anyone please help
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > > > > > parshant.kumar@indiamart.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > We have implemented collapse queries in place of
> > grouped
> > > > > > queries
> > > > > > > on
> > > > > > > > > our
> > > > > > > > > > > > production solr. As mentioned in solr documentation
> > > > collapse
> > > > > > > > queries
> > > > > > > > > > are
> > > > > > > > > > > > recommended in place of grouped queries in terms of
> > > > > > performance .
> > > > > > > > But
> > > > > > > > > > > after
> > > > > > > > > > > > switching to collapsed queries from grouped queries
> > > > response
> > > > > > time
> > > > > > > > of
> > > > > > > > > > > > queries have increased. This is unexpected behaviour,
> > the
> > > > > > > response
> > > > > > > > > time
> > > > > > > > > > > > should have been improved but results are opposites.
> > > > > > > > > > > > Please someone help why response time is increased
> for
> > > > > > collapsed
> > > > > > > > > > queries.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Parshant Kumar
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
--
Re: Increase in response time in case of collapse queries.
Posted by Joel Bernstein <jo...@gmail.com>.
Collapse is designed to outperform grouping in the following scenario:
There is high cardinality on the group field and group.ngroups is needed.
If either of these conditions is not satisfied grouping will typically be
faster.
You will need to provide some more information about your setup to get an
answer to the collapse performance question.
1) What are the response times for both methods. Saying one is faster is
not specific enough.
2) What is the cardinality of the collapse field, saying it's high is not
specific enough. What is the actual cardinality?
3) Is ngroups used in the grouping query.
Joel Bernstein
http://joelsolr.blogspot.com/
On Mon, Mar 8, 2021 at 11:30 AM Gajendra Dadheech <ga...@gmail.com>
wrote:
> @prashant Florin means to put null for parentglusrid in documents where
> this field-value is only present in one document [Group has only one
> document]. and then use nullPolicy to include/expand.
>
>
>
> On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
> <pa...@indiamart.com.invalid> wrote:
>
> > client should set to null the field if it's unique.
> >
> > @florin @Gajendra can you please explain more .I am not clear how to
> > perform this.
> >
> > On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <ba...@gmail.com>
> wrote:
> >
> > > @Gajendra Our response time dropped by 36% and our rps increased with
> > 27%.
> > >
> > > You have to reindex the core and the client should set to null the
> field
> > if
> > > it's unique.
> > >
> > > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> > > <pa...@indiamart.com.invalid> a scris:
> > >
> > > > How can we make group_field null? Using nullPolicy=expand ?
> > > >
> > > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <ba...@gmail.com>
> > > wrote:
> > > >
> > > > > We improved the performance of collapse by making the group_field
> > null
> > > > for
> > > > > the documents that have an unique value for group_field. This might
> > > help/
> > > > >
> > > > >
> > > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > > > <pa...@indiamart.com.invalid> a scris:
> > > > >
> > > > > > yes,group_field is having high cardinality.
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Parshant Kumar
> > > > > >
> > > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <
> babesflorin@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Your group_field has a high cardinality?
> > > > > > > Thanks,
> > > > > > > Florin Babes
> > > > > > >
> > > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > > >
> > > > > > > > Hi florin,
> > > > > > > >
> > > > > > > > I am using below.
> > > > > > > >
> > > > > > > > 1) fq={!collapse field=parentglusrid}
> > > > > > > > 2) expand.rows=4
> > > > > > > > 3) expand=true
> > > > > > > >
> > > > > > > > Size of index is around 100GB.
> > > > > > > > Solr version is 6.5
> > > > > > > >
> > > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > > babesflorin@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > > First let's call the field you collapse on group_field
> > > > > > > > > If group_field has a high cardinality you should make
> > > group_field
> > > > > > null
> > > > > > > > for
> > > > > > > > > those documents that have a unique group_field and set
> > > > > > > nullPolicy=expand.
> > > > > > > > > By doing that solr will use less memory for it's internal
> > maps
> > > > (so
> > > > > > > faster
> > > > > > > > > gc) and the head selecting will be faster.
> > > > > > > > > What is your head selecting strategy? Can you share your fq
> > > which
> > > > > you
> > > > > > > use
> > > > > > > > > for collapsing?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Florin Babes
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > > > > >
> > > > > > > > > > anyone please help
> > > > > > > > > >
> > > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > > > > parshant.kumar@indiamart.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > We have implemented collapse queries in place of
> grouped
> > > > > queries
> > > > > > on
> > > > > > > > our
> > > > > > > > > > > production solr. As mentioned in solr documentation
> > > collapse
> > > > > > > queries
> > > > > > > > > are
> > > > > > > > > > > recommended in place of grouped queries in terms of
> > > > > performance .
> > > > > > > But
> > > > > > > > > > after
> > > > > > > > > > > switching to collapsed queries from grouped queries
> > > response
> > > > > time
> > > > > > > of
> > > > > > > > > > > queries have increased. This is unexpected behaviour,
> the
> > > > > > response
> > > > > > > > time
> > > > > > > > > > > should have been improved but results are opposites.
> > > > > > > > > > > Please someone help why response time is increased for
> > > > > collapsed
> > > > > > > > > queries.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Parshant Kumar
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>
Re: Increase in response time in case of collapse queries.
Posted by Gajendra Dadheech <ga...@gmail.com>.
@prashant Florin means to put null for parentglusrid in documents where
this field-value is only present in one document [Group has only one
document]. and then use nullPolicy to include/expand.
On Mon, Mar 8, 2021 at 6:55 PM Parshant Kumar
<pa...@indiamart.com.invalid> wrote:
> client should set to null the field if it's unique.
>
> @florin @Gajendra can you please explain more .I am not clear how to
> perform this.
>
> On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <ba...@gmail.com> wrote:
>
> > @Gajendra Our response time dropped by 36% and our rps increased with
> 27%.
> >
> > You have to reindex the core and the client should set to null the field
> if
> > it's unique.
> >
> > În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> > <pa...@indiamart.com.invalid> a scris:
> >
> > > How can we make group_field null? Using nullPolicy=expand ?
> > >
> > > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <ba...@gmail.com>
> > wrote:
> > >
> > > > We improved the performance of collapse by making the group_field
> null
> > > for
> > > > the documents that have an unique value for group_field. This might
> > help/
> > > >
> > > >
> > > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > > <pa...@indiamart.com.invalid> a scris:
> > > >
> > > > > yes,group_field is having high cardinality.
> > > > >
> > > > >
> > > > > Thanks
> > > > > Parshant Kumar
> > > > >
> > > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <babesflorin@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Your group_field has a high cardinality?
> > > > > > Thanks,
> > > > > > Florin Babes
> > > > > >
> > > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > >
> > > > > > > Hi florin,
> > > > > > >
> > > > > > > I am using below.
> > > > > > >
> > > > > > > 1) fq={!collapse field=parentglusrid}
> > > > > > > 2) expand.rows=4
> > > > > > > 3) expand=true
> > > > > > >
> > > > > > > Size of index is around 100GB.
> > > > > > > Solr version is 6.5
> > > > > > >
> > > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> > babesflorin@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > > First let's call the field you collapse on group_field
> > > > > > > > If group_field has a high cardinality you should make
> > group_field
> > > > > null
> > > > > > > for
> > > > > > > > those documents that have a unique group_field and set
> > > > > > nullPolicy=expand.
> > > > > > > > By doing that solr will use less memory for it's internal
> maps
> > > (so
> > > > > > faster
> > > > > > > > gc) and the head selecting will be faster.
> > > > > > > > What is your head selecting strategy? Can you share your fq
> > which
> > > > you
> > > > > > use
> > > > > > > > for collapsing?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Florin Babes
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > > > >
> > > > > > > > > anyone please help
> > > > > > > > >
> > > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > > > parshant.kumar@indiamart.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > We have implemented collapse queries in place of grouped
> > > > queries
> > > > > on
> > > > > > > our
> > > > > > > > > > production solr. As mentioned in solr documentation
> > collapse
> > > > > > queries
> > > > > > > > are
> > > > > > > > > > recommended in place of grouped queries in terms of
> > > > performance .
> > > > > > But
> > > > > > > > > after
> > > > > > > > > > switching to collapsed queries from grouped queries
> > response
> > > > time
> > > > > > of
> > > > > > > > > > queries have increased. This is unexpected behaviour, the
> > > > > response
> > > > > > > time
> > > > > > > > > > should have been improved but results are opposites.
> > > > > > > > > > Please someone help why response time is increased for
> > > > collapsed
> > > > > > > > queries.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Parshant Kumar
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>
Re: Increase in response time in case of collapse queries.
Posted by Parshant Kumar <pa...@indiamart.com.INVALID>.
client should set to null the field if it's unique.
@florin @Gajendra can you please explain more .I am not clear how to
perform this.
On Mon, Mar 8, 2021 at 6:09 PM Florin Babes <ba...@gmail.com> wrote:
> @Gajendra Our response time dropped by 36% and our rps increased with 27%.
>
> You have to reindex the core and the client should set to null the field if
> it's unique.
>
> În lun., 8 mar. 2021 la 13:18, Parshant Kumar
> <pa...@indiamart.com.invalid> a scris:
>
> > How can we make group_field null? Using nullPolicy=expand ?
> >
> > On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <ba...@gmail.com>
> wrote:
> >
> > > We improved the performance of collapse by making the group_field null
> > for
> > > the documents that have an unique value for group_field. This might
> help/
> > >
> > >
> > > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > > <pa...@indiamart.com.invalid> a scris:
> > >
> > > > yes,group_field is having high cardinality.
> > > >
> > > >
> > > > Thanks
> > > > Parshant Kumar
> > > >
> > > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <ba...@gmail.com>
> > > wrote:
> > > >
> > > > > Your group_field has a high cardinality?
> > > > > Thanks,
> > > > > Florin Babes
> > > > >
> > > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > > <pa...@indiamart.com.invalid> a scris:
> > > > >
> > > > > > Hi florin,
> > > > > >
> > > > > > I am using below.
> > > > > >
> > > > > > 1) fq={!collapse field=parentglusrid}
> > > > > > 2) expand.rows=4
> > > > > > 3) expand=true
> > > > > >
> > > > > > Size of index is around 100GB.
> > > > > > Solr version is 6.5
> > > > > >
> > > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <
> babesflorin@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > > First let's call the field you collapse on group_field
> > > > > > > If group_field has a high cardinality you should make
> group_field
> > > > null
> > > > > > for
> > > > > > > those documents that have a unique group_field and set
> > > > > nullPolicy=expand.
> > > > > > > By doing that solr will use less memory for it's internal maps
> > (so
> > > > > faster
> > > > > > > gc) and the head selecting will be faster.
> > > > > > > What is your head selecting strategy? Can you share your fq
> which
> > > you
> > > > > use
> > > > > > > for collapsing?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Florin Babes
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > > >
> > > > > > > > anyone please help
> > > > > > > >
> > > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > > parshant.kumar@indiamart.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > We have implemented collapse queries in place of grouped
> > > queries
> > > > on
> > > > > > our
> > > > > > > > > production solr. As mentioned in solr documentation
> collapse
> > > > > queries
> > > > > > > are
> > > > > > > > > recommended in place of grouped queries in terms of
> > > performance .
> > > > > But
> > > > > > > > after
> > > > > > > > > switching to collapsed queries from grouped queries
> response
> > > time
> > > > > of
> > > > > > > > > queries have increased. This is unexpected behaviour, the
> > > > response
> > > > > > time
> > > > > > > > > should have been improved but results are opposites.
> > > > > > > > > Please someone help why response time is increased for
> > > collapsed
> > > > > > > queries.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Parshant Kumar
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>
--
Re: Increase in response time in case of collapse queries.
Posted by Florin Babes <ba...@gmail.com>.
@Gajendra Our response time dropped by 36% and our rps increased with 27%.
You have to reindex the core and the client should set to null the field if
it's unique.
În lun., 8 mar. 2021 la 13:18, Parshant Kumar
<pa...@indiamart.com.invalid> a scris:
> How can we make group_field null? Using nullPolicy=expand ?
>
> On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <ba...@gmail.com> wrote:
>
> > We improved the performance of collapse by making the group_field null
> for
> > the documents that have an unique value for group_field. This might help/
> >
> >
> > În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> > <pa...@indiamart.com.invalid> a scris:
> >
> > > yes,group_field is having high cardinality.
> > >
> > >
> > > Thanks
> > > Parshant Kumar
> > >
> > > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <ba...@gmail.com>
> > wrote:
> > >
> > > > Your group_field has a high cardinality?
> > > > Thanks,
> > > > Florin Babes
> > > >
> > > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > > <pa...@indiamart.com.invalid> a scris:
> > > >
> > > > > Hi florin,
> > > > >
> > > > > I am using below.
> > > > >
> > > > > 1) fq={!collapse field=parentglusrid}
> > > > > 2) expand.rows=4
> > > > > 3) expand=true
> > > > >
> > > > > Size of index is around 100GB.
> > > > > Solr version is 6.5
> > > > >
> > > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <babesflorin@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > > First let's call the field you collapse on group_field
> > > > > > If group_field has a high cardinality you should make group_field
> > > null
> > > > > for
> > > > > > those documents that have a unique group_field and set
> > > > nullPolicy=expand.
> > > > > > By doing that solr will use less memory for it's internal maps
> (so
> > > > faster
> > > > > > gc) and the head selecting will be faster.
> > > > > > What is your head selecting strategy? Can you share your fq which
> > you
> > > > use
> > > > > > for collapsing?
> > > > > >
> > > > > > Thanks,
> > > > > > Florin Babes
> > > > > >
> > > > > >
> > > > > >
> > > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > > <pa...@indiamart.com.invalid> a scris:
> > > > > >
> > > > > > > anyone please help
> > > > > > >
> > > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > > parshant.kumar@indiamart.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > We have implemented collapse queries in place of grouped
> > queries
> > > on
> > > > > our
> > > > > > > > production solr. As mentioned in solr documentation collapse
> > > > queries
> > > > > > are
> > > > > > > > recommended in place of grouped queries in terms of
> > performance .
> > > > But
> > > > > > > after
> > > > > > > > switching to collapsed queries from grouped queries response
> > time
> > > > of
> > > > > > > > queries have increased. This is unexpected behaviour, the
> > > response
> > > > > time
> > > > > > > > should have been improved but results are opposites.
> > > > > > > > Please someone help why response time is increased for
> > collapsed
> > > > > > queries.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Parshant Kumar
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>
Re: Increase in response time in case of collapse queries.
Posted by Parshant Kumar <pa...@indiamart.com.INVALID>.
How can we make group_field null? Using nullPolicy=expand ?
On Mon, Mar 8, 2021 at 4:41 PM Florin Babes <ba...@gmail.com> wrote:
> We improved the performance of collapse by making the group_field null for
> the documents that have an unique value for group_field. This might help/
>
>
> În lun., 8 mar. 2021 la 12:40, Parshant Kumar
> <pa...@indiamart.com.invalid> a scris:
>
> > yes,group_field is having high cardinality.
> >
> >
> > Thanks
> > Parshant Kumar
> >
> > On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <ba...@gmail.com>
> wrote:
> >
> > > Your group_field has a high cardinality?
> > > Thanks,
> > > Florin Babes
> > >
> > > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > > <pa...@indiamart.com.invalid> a scris:
> > >
> > > > Hi florin,
> > > >
> > > > I am using below.
> > > >
> > > > 1) fq={!collapse field=parentglusrid}
> > > > 2) expand.rows=4
> > > > 3) expand=true
> > > >
> > > > Size of index is around 100GB.
> > > > Solr version is 6.5
> > > >
> > > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <ba...@gmail.com>
> > > wrote:
> > > >
> > > > > Hello,
> > > > > First let's call the field you collapse on group_field
> > > > > If group_field has a high cardinality you should make group_field
> > null
> > > > for
> > > > > those documents that have a unique group_field and set
> > > nullPolicy=expand.
> > > > > By doing that solr will use less memory for it's internal maps (so
> > > faster
> > > > > gc) and the head selecting will be faster.
> > > > > What is your head selecting strategy? Can you share your fq which
> you
> > > use
> > > > > for collapsing?
> > > > >
> > > > > Thanks,
> > > > > Florin Babes
> > > > >
> > > > >
> > > > >
> > > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > > <pa...@indiamart.com.invalid> a scris:
> > > > >
> > > > > > anyone please help
> > > > > >
> > > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > > parshant.kumar@indiamart.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We have implemented collapse queries in place of grouped
> queries
> > on
> > > > our
> > > > > > > production solr. As mentioned in solr documentation collapse
> > > queries
> > > > > are
> > > > > > > recommended in place of grouped queries in terms of
> performance .
> > > But
> > > > > > after
> > > > > > > switching to collapsed queries from grouped queries response
> time
> > > of
> > > > > > > queries have increased. This is unexpected behaviour, the
> > response
> > > > time
> > > > > > > should have been improved but results are opposites.
> > > > > > > Please someone help why response time is increased for
> collapsed
> > > > > queries.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Parshant Kumar
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>
--
Re: Increase in response time in case of collapse queries.
Posted by Florin Babes <ba...@gmail.com>.
We improved the performance of collapse by making the group_field null for
the documents that have an unique value for group_field. This might help/
În lun., 8 mar. 2021 la 12:40, Parshant Kumar
<pa...@indiamart.com.invalid> a scris:
> yes,group_field is having high cardinality.
>
>
> Thanks
> Parshant Kumar
>
> On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <ba...@gmail.com> wrote:
>
> > Your group_field has a high cardinality?
> > Thanks,
> > Florin Babes
> >
> > În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> > <pa...@indiamart.com.invalid> a scris:
> >
> > > Hi florin,
> > >
> > > I am using below.
> > >
> > > 1) fq={!collapse field=parentglusrid}
> > > 2) expand.rows=4
> > > 3) expand=true
> > >
> > > Size of index is around 100GB.
> > > Solr version is 6.5
> > >
> > > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <ba...@gmail.com>
> > wrote:
> > >
> > > > Hello,
> > > > First let's call the field you collapse on group_field
> > > > If group_field has a high cardinality you should make group_field
> null
> > > for
> > > > those documents that have a unique group_field and set
> > nullPolicy=expand.
> > > > By doing that solr will use less memory for it's internal maps (so
> > faster
> > > > gc) and the head selecting will be faster.
> > > > What is your head selecting strategy? Can you share your fq which you
> > use
> > > > for collapsing?
> > > >
> > > > Thanks,
> > > > Florin Babes
> > > >
> > > >
> > > >
> > > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > > <pa...@indiamart.com.invalid> a scris:
> > > >
> > > > > anyone please help
> > > > >
> > > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > > parshant.kumar@indiamart.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We have implemented collapse queries in place of grouped queries
> on
> > > our
> > > > > > production solr. As mentioned in solr documentation collapse
> > queries
> > > > are
> > > > > > recommended in place of grouped queries in terms of performance .
> > But
> > > > > after
> > > > > > switching to collapsed queries from grouped queries response time
> > of
> > > > > > queries have increased. This is unexpected behaviour, the
> response
> > > time
> > > > > > should have been improved but results are opposites.
> > > > > > Please someone help why response time is increased for collapsed
> > > > queries.
> > > > > >
> > > > > > Thanks
> > > > > > Parshant Kumar
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>
Re: Increase in response time in case of collapse queries.
Posted by Parshant Kumar <pa...@indiamart.com.INVALID>.
yes,group_field is having high cardinality.
Thanks
Parshant Kumar
On Mon, Mar 8, 2021 at 4:06 PM Florin Babes <ba...@gmail.com> wrote:
> Your group_field has a high cardinality?
> Thanks,
> Florin Babes
>
> În lun., 8 mar. 2021 la 10:35, Parshant Kumar
> <pa...@indiamart.com.invalid> a scris:
>
> > Hi florin,
> >
> > I am using below.
> >
> > 1) fq={!collapse field=parentglusrid}
> > 2) expand.rows=4
> > 3) expand=true
> >
> > Size of index is around 100GB.
> > Solr version is 6.5
> >
> > On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <ba...@gmail.com>
> wrote:
> >
> > > Hello,
> > > First let's call the field you collapse on group_field
> > > If group_field has a high cardinality you should make group_field null
> > for
> > > those documents that have a unique group_field and set
> nullPolicy=expand.
> > > By doing that solr will use less memory for it's internal maps (so
> faster
> > > gc) and the head selecting will be faster.
> > > What is your head selecting strategy? Can you share your fq which you
> use
> > > for collapsing?
> > >
> > > Thanks,
> > > Florin Babes
> > >
> > >
> > >
> > > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > > <pa...@indiamart.com.invalid> a scris:
> > >
> > > > anyone please help
> > > >
> > > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > > parshant.kumar@indiamart.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We have implemented collapse queries in place of grouped queries on
> > our
> > > > > production solr. As mentioned in solr documentation collapse
> queries
> > > are
> > > > > recommended in place of grouped queries in terms of performance .
> But
> > > > after
> > > > > switching to collapsed queries from grouped queries response time
> of
> > > > > queries have increased. This is unexpected behaviour, the response
> > time
> > > > > should have been improved but results are opposites.
> > > > > Please someone help why response time is increased for collapsed
> > > queries.
> > > > >
> > > > > Thanks
> > > > > Parshant Kumar
> > > > >
> > > >
> > > > --
> > > >
> > > >
> > >
> >
> > --
> >
> >
>
--
Re: Increase in response time in case of collapse queries.
Posted by Florin Babes <ba...@gmail.com>.
Your group_field has a high cardinality?
Thanks,
Florin Babes
În lun., 8 mar. 2021 la 10:35, Parshant Kumar
<pa...@indiamart.com.invalid> a scris:
> Hi florin,
>
> I am using below.
>
> 1) fq={!collapse field=parentglusrid}
> 2) expand.rows=4
> 3) expand=true
>
> Size of index is around 100GB.
> Solr version is 6.5
>
> On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <ba...@gmail.com> wrote:
>
> > Hello,
> > First let's call the field you collapse on group_field
> > If group_field has a high cardinality you should make group_field null
> for
> > those documents that have a unique group_field and set nullPolicy=expand.
> > By doing that solr will use less memory for it's internal maps (so faster
> > gc) and the head selecting will be faster.
> > What is your head selecting strategy? Can you share your fq which you use
> > for collapsing?
> >
> > Thanks,
> > Florin Babes
> >
> >
> >
> > În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> > <pa...@indiamart.com.invalid> a scris:
> >
> > > anyone please help
> > >
> > > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > > parshant.kumar@indiamart.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > We have implemented collapse queries in place of grouped queries on
> our
> > > > production solr. As mentioned in solr documentation collapse queries
> > are
> > > > recommended in place of grouped queries in terms of performance . But
> > > after
> > > > switching to collapsed queries from grouped queries response time of
> > > > queries have increased. This is unexpected behaviour, the response
> time
> > > > should have been improved but results are opposites.
> > > > Please someone help why response time is increased for collapsed
> > queries.
> > > >
> > > > Thanks
> > > > Parshant Kumar
> > > >
> > >
> > > --
> > >
> > >
> >
>
> --
>
>
Re: Increase in response time in case of collapse queries.
Posted by Parshant Kumar <pa...@indiamart.com.INVALID>.
Hi florin,
I am using below.
1) fq={!collapse field=parentglusrid}
2) expand.rows=4
3) expand=true
Size of index is around 100GB.
Solr version is 6.5
On Mon, Mar 8, 2021 at 1:46 PM Florin Babes <ba...@gmail.com> wrote:
> Hello,
> First let's call the field you collapse on group_field
> If group_field has a high cardinality you should make group_field null for
> those documents that have a unique group_field and set nullPolicy=expand.
> By doing that solr will use less memory for it's internal maps (so faster
> gc) and the head selecting will be faster.
> What is your head selecting strategy? Can you share your fq which you use
> for collapsing?
>
> Thanks,
> Florin Babes
>
>
>
> În lun., 8 mar. 2021 la 06:44, Parshant Kumar
> <pa...@indiamart.com.invalid> a scris:
>
> > anyone please help
> >
> > On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> > parshant.kumar@indiamart.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > We have implemented collapse queries in place of grouped queries on our
> > > production solr. As mentioned in solr documentation collapse queries
> are
> > > recommended in place of grouped queries in terms of performance . But
> > after
> > > switching to collapsed queries from grouped queries response time of
> > > queries have increased. This is unexpected behaviour, the response time
> > > should have been improved but results are opposites.
> > > Please someone help why response time is increased for collapsed
> queries.
> > >
> > > Thanks
> > > Parshant Kumar
> > >
> >
> > --
> >
> >
>
--
Re: Increase in response time in case of collapse queries.
Posted by Florin Babes <ba...@gmail.com>.
Hello,
First let's call the field you collapse on group_field
If group_field has a high cardinality you should make group_field null for
those documents that have a unique group_field and set nullPolicy=expand.
By doing that solr will use less memory for it's internal maps (so faster
gc) and the head selecting will be faster.
What is your head selecting strategy? Can you share your fq which you use
for collapsing?
Thanks,
Florin Babes
În lun., 8 mar. 2021 la 06:44, Parshant Kumar
<pa...@indiamart.com.invalid> a scris:
> anyone please help
>
> On Wed, Mar 3, 2021 at 4:55 PM Parshant Kumar <
> parshant.kumar@indiamart.com>
> wrote:
>
> > Hi all,
> >
> > We have implemented collapse queries in place of grouped queries on our
> > production solr. As mentioned in solr documentation collapse queries are
> > recommended in place of grouped queries in terms of performance . But
> after
> > switching to collapsed queries from grouped queries response time of
> > queries have increased. This is unexpected behaviour, the response time
> > should have been improved but results are opposites.
> > Please someone help why response time is increased for collapsed queries.
> >
> > Thanks
> > Parshant Kumar
> >
>
> --
>
>