You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Brian Narsi <bn...@gmail.com> on 2016/02/11 16:14:17 UTC
Select distinct records
I am trying to select distinct records from a collection. (I need distinct
name and corresponding id)
I have tried using grouping and group format of simple but that takes a
long time to execute and sometimes runs into out of memory exception.
Another limitation seems to be that total number of groups are not returned.
Is there another faster and more efficient way to do this?
Thank you
Re: Select distinct records
Posted by Brian Narsi <bn...@gmail.com>.
In order to use the Collapsing feature I will need to use Document Routing
to co-locate related documents in the same shard in SolrCloud. What are the
advantages and disadvantages of Document Routing?
Thanks,
On Thu, Feb 11, 2016 at 12:54 PM, Joel Bernstein <jo...@gmail.com> wrote:
> Yeah that would be the reason. If you want distributed unique capabilities,
> then you might want to start testing out 6.0. Aside from SELECT DISTINCT
> queries, you also have a much more mature Streaming Expression library
> which supports the unique operation.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Feb 11, 2016 at 12:28 PM, Brian Narsi <bn...@gmail.com> wrote:
>
> > Ok I see that Collapsing features requires documents to be co-located in
> > the same shard in SolrCloud.
> >
> > Could that be a reason for duplication?
> >
> > On Thu, Feb 11, 2016 at 11:09 AM, Joel Bernstein <jo...@gmail.com>
> > wrote:
> >
> > > The CollapsingQParserPlugin shouldn't have duplicates in the result
> set.
> > > Can you provide the details?
> > >
> > > Joel Bernstein
> > > http://joelsolr.blogspot.com/
> > >
> > > On Thu, Feb 11, 2016 at 12:02 PM, Brian Narsi <bn...@gmail.com>
> > wrote:
> > >
> > > > I have tried to use the Collapsing feature but it appears that it
> > leaves
> > > > duplicated records in the result set.
> > > >
> > > > Is that expected? Or any suggestions on working around it?
> > > >
> > > > Thanks
> > > >
> > > > On Thu, Feb 11, 2016 at 9:30 AM, Brian Narsi <bn...@gmail.com>
> > wrote:
> > > >
> > > > > I am using
> > > > >
> > > > > Solr 5.1.0
> > > > >
> > > > > On Thu, Feb 11, 2016 at 9:19 AM, Binoy Dalal <
> binoydalal93@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > >> What version of Solr are you using?
> > > > >> Have you taken a look at the Collapsing Query Parser. It basically
> > > > >> performs
> > > > >> the same functions as grouping but is much more efficient at doing
> > it.
> > > > >> Take a look here:
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
> > > > >>
> > > > >> On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > I am trying to select distinct records from a collection. (I
> need
> > > > >> distinct
> > > > >> > name and corresponding id)
> > > > >> >
> > > > >> > I have tried using grouping and group format of simple but that
> > > takes
> > > > a
> > > > >> > long time to execute and sometimes runs into out of memory
> > > exception.
> > > > >> > Another limitation seems to be that total number of groups are
> not
> > > > >> > returned.
> > > > >> >
> > > > >> > Is there another faster and more efficient way to do this?
> > > > >> >
> > > > >> > Thank you
> > > > >> >
> > > > >> --
> > > > >> Regards,
> > > > >> Binoy Dalal
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Select distinct records
Posted by Joel Bernstein <jo...@gmail.com>.
Yeah that would be the reason. If you want distributed unique capabilities,
then you might want to start testing out 6.0. Aside from SELECT DISTINCT
queries, you also have a much more mature Streaming Expression library
which supports the unique operation.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Feb 11, 2016 at 12:28 PM, Brian Narsi <bn...@gmail.com> wrote:
> Ok I see that Collapsing features requires documents to be co-located in
> the same shard in SolrCloud.
>
> Could that be a reason for duplication?
>
> On Thu, Feb 11, 2016 at 11:09 AM, Joel Bernstein <jo...@gmail.com>
> wrote:
>
> > The CollapsingQParserPlugin shouldn't have duplicates in the result set.
> > Can you provide the details?
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Thu, Feb 11, 2016 at 12:02 PM, Brian Narsi <bn...@gmail.com>
> wrote:
> >
> > > I have tried to use the Collapsing feature but it appears that it
> leaves
> > > duplicated records in the result set.
> > >
> > > Is that expected? Or any suggestions on working around it?
> > >
> > > Thanks
> > >
> > > On Thu, Feb 11, 2016 at 9:30 AM, Brian Narsi <bn...@gmail.com>
> wrote:
> > >
> > > > I am using
> > > >
> > > > Solr 5.1.0
> > > >
> > > > On Thu, Feb 11, 2016 at 9:19 AM, Binoy Dalal <binoydalal93@gmail.com
> >
> > > > wrote:
> > > >
> > > >> What version of Solr are you using?
> > > >> Have you taken a look at the Collapsing Query Parser. It basically
> > > >> performs
> > > >> the same functions as grouping but is much more efficient at doing
> it.
> > > >> Take a look here:
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
> > > >>
> > > >> On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com>
> > wrote:
> > > >>
> > > >> > I am trying to select distinct records from a collection. (I need
> > > >> distinct
> > > >> > name and corresponding id)
> > > >> >
> > > >> > I have tried using grouping and group format of simple but that
> > takes
> > > a
> > > >> > long time to execute and sometimes runs into out of memory
> > exception.
> > > >> > Another limitation seems to be that total number of groups are not
> > > >> > returned.
> > > >> >
> > > >> > Is there another faster and more efficient way to do this?
> > > >> >
> > > >> > Thank you
> > > >> >
> > > >> --
> > > >> Regards,
> > > >> Binoy Dalal
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Select distinct records
Posted by Brian Narsi <bn...@gmail.com>.
Ok I see that Collapsing features requires documents to be co-located in
the same shard in SolrCloud.
Could that be a reason for duplication?
On Thu, Feb 11, 2016 at 11:09 AM, Joel Bernstein <jo...@gmail.com> wrote:
> The CollapsingQParserPlugin shouldn't have duplicates in the result set.
> Can you provide the details?
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Feb 11, 2016 at 12:02 PM, Brian Narsi <bn...@gmail.com> wrote:
>
> > I have tried to use the Collapsing feature but it appears that it leaves
> > duplicated records in the result set.
> >
> > Is that expected? Or any suggestions on working around it?
> >
> > Thanks
> >
> > On Thu, Feb 11, 2016 at 9:30 AM, Brian Narsi <bn...@gmail.com> wrote:
> >
> > > I am using
> > >
> > > Solr 5.1.0
> > >
> > > On Thu, Feb 11, 2016 at 9:19 AM, Binoy Dalal <bi...@gmail.com>
> > > wrote:
> > >
> > >> What version of Solr are you using?
> > >> Have you taken a look at the Collapsing Query Parser. It basically
> > >> performs
> > >> the same functions as grouping but is much more efficient at doing it.
> > >> Take a look here:
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
> > >>
> > >> On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com>
> wrote:
> > >>
> > >> > I am trying to select distinct records from a collection. (I need
> > >> distinct
> > >> > name and corresponding id)
> > >> >
> > >> > I have tried using grouping and group format of simple but that
> takes
> > a
> > >> > long time to execute and sometimes runs into out of memory
> exception.
> > >> > Another limitation seems to be that total number of groups are not
> > >> > returned.
> > >> >
> > >> > Is there another faster and more efficient way to do this?
> > >> >
> > >> > Thank you
> > >> >
> > >> --
> > >> Regards,
> > >> Binoy Dalal
> > >>
> > >
> > >
> >
>
Re: Select distinct records
Posted by Joel Bernstein <jo...@gmail.com>.
The CollapsingQParserPlugin shouldn't have duplicates in the result set.
Can you provide the details?
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Feb 11, 2016 at 12:02 PM, Brian Narsi <bn...@gmail.com> wrote:
> I have tried to use the Collapsing feature but it appears that it leaves
> duplicated records in the result set.
>
> Is that expected? Or any suggestions on working around it?
>
> Thanks
>
> On Thu, Feb 11, 2016 at 9:30 AM, Brian Narsi <bn...@gmail.com> wrote:
>
> > I am using
> >
> > Solr 5.1.0
> >
> > On Thu, Feb 11, 2016 at 9:19 AM, Binoy Dalal <bi...@gmail.com>
> > wrote:
> >
> >> What version of Solr are you using?
> >> Have you taken a look at the Collapsing Query Parser. It basically
> >> performs
> >> the same functions as grouping but is much more efficient at doing it.
> >> Take a look here:
> >>
> >>
> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
> >>
> >> On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com> wrote:
> >>
> >> > I am trying to select distinct records from a collection. (I need
> >> distinct
> >> > name and corresponding id)
> >> >
> >> > I have tried using grouping and group format of simple but that takes
> a
> >> > long time to execute and sometimes runs into out of memory exception.
> >> > Another limitation seems to be that total number of groups are not
> >> > returned.
> >> >
> >> > Is there another faster and more efficient way to do this?
> >> >
> >> > Thank you
> >> >
> >> --
> >> Regards,
> >> Binoy Dalal
> >>
> >
> >
>
Re: Select distinct records
Posted by Brian Narsi <bn...@gmail.com>.
I have tried to use the Collapsing feature but it appears that it leaves
duplicated records in the result set.
Is that expected? Or any suggestions on working around it?
Thanks
On Thu, Feb 11, 2016 at 9:30 AM, Brian Narsi <bn...@gmail.com> wrote:
> I am using
>
> Solr 5.1.0
>
> On Thu, Feb 11, 2016 at 9:19 AM, Binoy Dalal <bi...@gmail.com>
> wrote:
>
>> What version of Solr are you using?
>> Have you taken a look at the Collapsing Query Parser. It basically
>> performs
>> the same functions as grouping but is much more efficient at doing it.
>> Take a look here:
>>
>> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
>>
>> On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com> wrote:
>>
>> > I am trying to select distinct records from a collection. (I need
>> distinct
>> > name and corresponding id)
>> >
>> > I have tried using grouping and group format of simple but that takes a
>> > long time to execute and sometimes runs into out of memory exception.
>> > Another limitation seems to be that total number of groups are not
>> > returned.
>> >
>> > Is there another faster and more efficient way to do this?
>> >
>> > Thank you
>> >
>> --
>> Regards,
>> Binoy Dalal
>>
>
>
Re: Select distinct records
Posted by Joel Bernstein <jo...@gmail.com>.
Solr 6.0 supports SELECT DISTINCT (SQL) queries. You can even choose
between a MapReduce implementation and a Json Facet implementation. The
MapReduce Implementation supports extremely high cardinality for the
distinct fields. Json Facet implementation supports lower cardinality but
high QPS.
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Feb 11, 2016 at 10:30 AM, Brian Narsi <bn...@gmail.com> wrote:
> I am using
>
> Solr 5.1.0
>
> On Thu, Feb 11, 2016 at 9:19 AM, Binoy Dalal <bi...@gmail.com>
> wrote:
>
> > What version of Solr are you using?
> > Have you taken a look at the Collapsing Query Parser. It basically
> performs
> > the same functions as grouping but is much more efficient at doing it.
> > Take a look here:
> >
> >
> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
> >
> > On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com> wrote:
> >
> > > I am trying to select distinct records from a collection. (I need
> > distinct
> > > name and corresponding id)
> > >
> > > I have tried using grouping and group format of simple but that takes a
> > > long time to execute and sometimes runs into out of memory exception.
> > > Another limitation seems to be that total number of groups are not
> > > returned.
> > >
> > > Is there another faster and more efficient way to do this?
> > >
> > > Thank you
> > >
> > --
> > Regards,
> > Binoy Dalal
> >
>
Re: Select distinct records
Posted by Brian Narsi <bn...@gmail.com>.
I am using
Solr 5.1.0
On Thu, Feb 11, 2016 at 9:19 AM, Binoy Dalal <bi...@gmail.com> wrote:
> What version of Solr are you using?
> Have you taken a look at the Collapsing Query Parser. It basically performs
> the same functions as grouping but is much more efficient at doing it.
> Take a look here:
>
> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
>
> On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com> wrote:
>
> > I am trying to select distinct records from a collection. (I need
> distinct
> > name and corresponding id)
> >
> > I have tried using grouping and group format of simple but that takes a
> > long time to execute and sometimes runs into out of memory exception.
> > Another limitation seems to be that total number of groups are not
> > returned.
> >
> > Is there another faster and more efficient way to do this?
> >
> > Thank you
> >
> --
> Regards,
> Binoy Dalal
>
Re: Select distinct records
Posted by Binoy Dalal <bi...@gmail.com>.
What version of Solr are you using?
Have you taken a look at the Collapsing Query Parser. It basically performs
the same functions as grouping but is much more efficient at doing it.
Take a look here:
https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
On Thu, Feb 11, 2016 at 8:44 PM Brian Narsi <bn...@gmail.com> wrote:
> I am trying to select distinct records from a collection. (I need distinct
> name and corresponding id)
>
> I have tried using grouping and group format of simple but that takes a
> long time to execute and sometimes runs into out of memory exception.
> Another limitation seems to be that total number of groups are not
> returned.
>
> Is there another faster and more efficient way to do this?
>
> Thank you
>
--
Regards,
Binoy Dalal