You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Matt Mitchell <go...@gmail.com> on 2009/05/16 00:52:48 UTC
grouping response docs together
Is there a built-in mechanism for grouping similar documents together in the
response? I'd like to make it look like there is only one document with
multiple "hits".
Matt
Re: grouping response docs together
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Matt - what you are describing is about search-time grouping a la SOLR-236.
Deduplication stuff I mention is for index-time near-duplicate detection.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Matt Mitchell <go...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Tuesday, May 26, 2009 1:40:01 PM
> Subject: Re: grouping response docs together
>
> Thanks Otis. I'll give the dedup a test drive today.
>
> I'll explain what I'm trying to do a little better though because I don't
> think I have yet!
>
> So, I'm indexing an XML file. There are different "sections" in the XML
> file. Each of those sections gets a solr doc (the xml text-only is indexed).
> Each solr doc also has a field to specify the source filename. What I'd like
> to have happen is, when I do a search, I want my search results to combine
> all documents that have the same filename... I want to "group by" filename
> if that makes sense. Or at the very least, show only one and indicate that
> there are more.
>
> Matt
>
> On Tue, May 26, 2009 at 12:58 PM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com> wrote:
>
> >
> > Matt,
> >
> > The Deduplication feature in Solr does support near-duplicate scenario. It
> > comes with a few components to help you detect near-duplicates, and you
> > should be able to write a custom near-dupe detection component and plug it
> > in.
> >
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> > > From: Matt Mitchell
> > > To: solr-user@lucene.apache.org
> > > Sent: Monday, May 25, 2009 3:30:42 PM
> > > Subject: Re: grouping response docs together
> > >
> > > Thanks guys. I looked at the dedup stuff, but the documents I'm adding
> > > aren't really duplicates. They're very similar, but different.
> > >
> > > I checked out the field collapsing feature patch, applied the patch but
> > > can't get it to build successfully. Will this patch work with a nightly
> > > build?
> > >
> > > Thanks!
> > >
> > > On Fri, May 15, 2009 at 7:47 PM, Otis Gospodnetic <
> > > otis_gospodnetic@yahoo.com> wrote:
> > >
> > > >
> > > > Matt - you may also want to detect near duplicates at index time:
> > > >
> > > > http://wiki.apache.org/solr/Deduplication
> > > >
> > > > Otis
> > > > --
> > > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > > >
> > > >
> > > >
> > > > ----- Original Message ----
> > > > > From: Matt Mitchell
> > > > > To: solr-user@lucene.apache.org
> > > > > Sent: Friday, May 15, 2009 6:52:48 PM
> > > > > Subject: grouping response docs together
> > > > >
> > > > > Is there a built-in mechanism for grouping similar documents together
> > in
> > > > the
> > > > > response? I'd like to make it look like there is only one document
> > with
> > > > > multiple "hits".
> > > > >
> > > > > Matt
> > > >
> > > >
> >
> >
Re: grouping response docs together
Posted by Matt Mitchell <go...@gmail.com>.
Thanks Otis. I'll give the dedup a test drive today.
I'll explain what I'm trying to do a little better though because I don't
think I have yet!
So, I'm indexing an XML file. There are different "sections" in the XML
file. Each of those sections gets a solr doc (the xml text-only is indexed).
Each solr doc also has a field to specify the source filename. What I'd like
to have happen is, when I do a search, I want my search results to combine
all documents that have the same filename... I want to "group by" filename
if that makes sense. Or at the very least, show only one and indicate that
there are more.
Matt
On Tue, May 26, 2009 at 12:58 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
>
> Matt,
>
> The Deduplication feature in Solr does support near-duplicate scenario. It
> comes with a few components to help you detect near-duplicates, and you
> should be able to write a custom near-dupe detection component and plug it
> in.
>
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: Matt Mitchell <go...@gmail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Monday, May 25, 2009 3:30:42 PM
> > Subject: Re: grouping response docs together
> >
> > Thanks guys. I looked at the dedup stuff, but the documents I'm adding
> > aren't really duplicates. They're very similar, but different.
> >
> > I checked out the field collapsing feature patch, applied the patch but
> > can't get it to build successfully. Will this patch work with a nightly
> > build?
> >
> > Thanks!
> >
> > On Fri, May 15, 2009 at 7:47 PM, Otis Gospodnetic <
> > otis_gospodnetic@yahoo.com> wrote:
> >
> > >
> > > Matt - you may also want to detect near duplicates at index time:
> > >
> > > http://wiki.apache.org/solr/Deduplication
> > >
> > > Otis
> > > --
> > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> > >
> > >
> > >
> > > ----- Original Message ----
> > > > From: Matt Mitchell
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Friday, May 15, 2009 6:52:48 PM
> > > > Subject: grouping response docs together
> > > >
> > > > Is there a built-in mechanism for grouping similar documents together
> in
> > > the
> > > > response? I'd like to make it look like there is only one document
> with
> > > > multiple "hits".
> > > >
> > > > Matt
> > >
> > >
>
>
Re: grouping response docs together
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Matt,
The Deduplication feature in Solr does support near-duplicate scenario. It comes with a few components to help you detect near-duplicates, and you should be able to write a custom near-dupe detection component and plug it in.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Matt Mitchell <go...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Monday, May 25, 2009 3:30:42 PM
> Subject: Re: grouping response docs together
>
> Thanks guys. I looked at the dedup stuff, but the documents I'm adding
> aren't really duplicates. They're very similar, but different.
>
> I checked out the field collapsing feature patch, applied the patch but
> can't get it to build successfully. Will this patch work with a nightly
> build?
>
> Thanks!
>
> On Fri, May 15, 2009 at 7:47 PM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com> wrote:
>
> >
> > Matt - you may also want to detect near duplicates at index time:
> >
> > http://wiki.apache.org/solr/Deduplication
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> > > From: Matt Mitchell
> > > To: solr-user@lucene.apache.org
> > > Sent: Friday, May 15, 2009 6:52:48 PM
> > > Subject: grouping response docs together
> > >
> > > Is there a built-in mechanism for grouping similar documents together in
> > the
> > > response? I'd like to make it look like there is only one document with
> > > multiple "hits".
> > >
> > > Matt
> >
> >
Re: grouping response docs together
Posted by Matt Mitchell <go...@gmail.com>.
Thomas,
Here's what I get after patching nightly build (yesterday) and running "ant
test"
compileTests:
[javac] Compiling 1 source file to
/Users/mwm4n/Downloads/apache-solr-nightly/build/tests
[javac]
/Users/mwm4n/Downloads/apache-solr-nightly/src/test/org/apache/solr/search/TestDocSet.java:138:
checkEqual(org.apache.lucene.util.OpenBitSet,org.apache.solr.search.DocSet)
in org.apache.solr.search.TestDocSet cannot be applied to
(org.apache.solr.search.DocSet,org.apache.solr.search.DocSet)
[javac] checkEqual(a1,
NegatedDocSet.negation(NegatedDocSet.negation(b1)));
[javac] ^
[javac] 1 error
Matt
On Mon, May 25, 2009 at 7:59 PM, Matt Mitchell <go...@gmail.com> wrote:
> Hi Thomas,
>
> In a 5-24-09 nightly build, I applied the patch:
>
> cd apache-solr-nightly
>
> patch -p0 < ~/Projects/apache-solr-patches/SOLR-236_collapsing.patch
> patching file src/common/org/apache/solr/common/params/CollapseParams.java
> patching file
> src/java/org/apache/solr/handler/component/CollapseComponent.java
> patching file src/java/org/apache/solr/search/CollapseFilter.java
> patching file src/java/org/apache/solr/search/NegatedDocSet.java
> patching file src/java/org/apache/solr/search/SolrIndexSearcher.java
> Hunk #1 succeeded at 1444 (offset -39 lines).
> patching file src/test/org/apache/solr/search/TestDocSet.java
> Hunk #1 succeeded at 134 (offset 42 lines).
>
> ... and got this when running "ant dist"
>
> docs:
> [mkdir] Created dir:
> /Users/mwm4n/Downloads/apache-solr-nightly/contrib/javascript/dist/doc
> [java] Exception in thread "main" java.lang.NoClassDefFoundError:
> org/mozilla/javascript/tools/shell/Main
> [java] at JsRun.main(Unknown Source)
>
> BUILD FAILED
> /Users/mwm4n/Downloads/apache-solr-nightly/common-build.xml:338: The
> following error occurred while executing this line:
> /Users/mwm4n/Downloads/apache-solr-nightly/common-build.xml:215: The
> following error occurred while executing this line:
> /Users/mwm4n/Downloads/apache-solr-nightly/contrib/javascript/build.xml:74:
> Java returned: 1
>
> Not sure what any of that means, but the "ant dist" task worked fine before
> the patch. Any ideas?
>
> Thanks,
>
> Matt
>
>
> On Mon, May 25, 2009 at 3:59 PM, Thomas Traeger <t....@kabuco.de>wrote:
>
>> Hello Matt,
>>
>> the patch should work with trunk and after a small fix with 1.3 too (see
>> my comment in SOLR-236). I just made a successful build to be sure.
>>
>> Do you see any error messages?
>>
>> Thomas
>>
>> Matt Mitchell schrieb:
>>
>> Thanks guys. I looked at the dedup stuff, but the documents I'm adding
>>> aren't really duplicates. They're very similar, but different.
>>>
>>> I checked out the field collapsing feature patch, applied the patch but
>>> can't get it to build successfully. Will this patch work with a nightly
>>> build?
>>>
>>> Thanks!
>>>
>>> On Fri, May 15, 2009 at 7:47 PM, Otis Gospodnetic <
>>> otis_gospodnetic@yahoo.com> wrote:
>>>
>>> Matt - you may also want to detect near duplicates at index time:
>>>>
>>>> http://wiki.apache.org/solr/Deduplication
>>>>
>>>> Otis
>>>> --
>>>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>>>
>>>>
>>>>
>>>> ----- Original Message ----
>>>>
>>>>> From: Matt Mitchell <go...@gmail.com>
>>>>> To: solr-user@lucene.apache.org
>>>>> Sent: Friday, May 15, 2009 6:52:48 PM
>>>>> Subject: grouping response docs together
>>>>>
>>>>> Is there a built-in mechanism for grouping similar documents together
>>>>> in
>>>>>
>>>> the
>>>>
>>>>> response? I'd like to make it look like there is only one document with
>>>>> multiple "hits".
>>>>>
>>>>> Matt
>>>>>
>>>>
>>>>
>>>
>>
>
Re: grouping response docs together
Posted by Matt Mitchell <go...@gmail.com>.
Hi Thomas,
In a 5-24-09 nightly build, I applied the patch:
cd apache-solr-nightly
patch -p0 < ~/Projects/apache-solr-patches/SOLR-236_collapsing.patch
patching file src/common/org/apache/solr/common/params/CollapseParams.java
patching file
src/java/org/apache/solr/handler/component/CollapseComponent.java
patching file src/java/org/apache/solr/search/CollapseFilter.java
patching file src/java/org/apache/solr/search/NegatedDocSet.java
patching file src/java/org/apache/solr/search/SolrIndexSearcher.java
Hunk #1 succeeded at 1444 (offset -39 lines).
patching file src/test/org/apache/solr/search/TestDocSet.java
Hunk #1 succeeded at 134 (offset 42 lines).
... and got this when running "ant dist"
docs:
[mkdir] Created dir:
/Users/mwm4n/Downloads/apache-solr-nightly/contrib/javascript/dist/doc
[java] Exception in thread "main" java.lang.NoClassDefFoundError:
org/mozilla/javascript/tools/shell/Main
[java] at JsRun.main(Unknown Source)
BUILD FAILED
/Users/mwm4n/Downloads/apache-solr-nightly/common-build.xml:338: The
following error occurred while executing this line:
/Users/mwm4n/Downloads/apache-solr-nightly/common-build.xml:215: The
following error occurred while executing this line:
/Users/mwm4n/Downloads/apache-solr-nightly/contrib/javascript/build.xml:74:
Java returned: 1
Not sure what any of that means, but the "ant dist" task worked fine before
the patch. Any ideas?
Thanks,
Matt
On Mon, May 25, 2009 at 3:59 PM, Thomas Traeger <t....@kabuco.de> wrote:
> Hello Matt,
>
> the patch should work with trunk and after a small fix with 1.3 too (see
> my comment in SOLR-236). I just made a successful build to be sure.
>
> Do you see any error messages?
>
> Thomas
>
> Matt Mitchell schrieb:
>
> Thanks guys. I looked at the dedup stuff, but the documents I'm adding
>> aren't really duplicates. They're very similar, but different.
>>
>> I checked out the field collapsing feature patch, applied the patch but
>> can't get it to build successfully. Will this patch work with a nightly
>> build?
>>
>> Thanks!
>>
>> On Fri, May 15, 2009 at 7:47 PM, Otis Gospodnetic <
>> otis_gospodnetic@yahoo.com> wrote:
>>
>> Matt - you may also want to detect near duplicates at index time:
>>>
>>> http://wiki.apache.org/solr/Deduplication
>>>
>>> Otis
>>> --
>>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>>
>>>
>>>
>>> ----- Original Message ----
>>>
>>>> From: Matt Mitchell <go...@gmail.com>
>>>> To: solr-user@lucene.apache.org
>>>> Sent: Friday, May 15, 2009 6:52:48 PM
>>>> Subject: grouping response docs together
>>>>
>>>> Is there a built-in mechanism for grouping similar documents together in
>>>>
>>> the
>>>
>>>> response? I'd like to make it look like there is only one document with
>>>> multiple "hits".
>>>>
>>>> Matt
>>>>
>>>
>>>
>>
>
Re: grouping response docs together
Posted by Thomas Traeger <t....@kabuco.de>.
Hello Matt,
the patch should work with trunk and after a small fix with 1.3 too (see
my comment in SOLR-236). I just made a successful build to be sure.
Do you see any error messages?
Thomas
Matt Mitchell schrieb:
> Thanks guys. I looked at the dedup stuff, but the documents I'm adding
> aren't really duplicates. They're very similar, but different.
>
> I checked out the field collapsing feature patch, applied the patch but
> can't get it to build successfully. Will this patch work with a nightly
> build?
>
> Thanks!
>
> On Fri, May 15, 2009 at 7:47 PM, Otis Gospodnetic <
> otis_gospodnetic@yahoo.com> wrote:
>
>> Matt - you may also want to detect near duplicates at index time:
>>
>> http://wiki.apache.org/solr/Deduplication
>>
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>>
>>
>>
>> ----- Original Message ----
>>> From: Matt Mitchell <go...@gmail.com>
>>> To: solr-user@lucene.apache.org
>>> Sent: Friday, May 15, 2009 6:52:48 PM
>>> Subject: grouping response docs together
>>>
>>> Is there a built-in mechanism for grouping similar documents together in
>> the
>>> response? I'd like to make it look like there is only one document with
>>> multiple "hits".
>>>
>>> Matt
>>
>
Re: grouping response docs together
Posted by Matt Mitchell <go...@gmail.com>.
Thanks guys. I looked at the dedup stuff, but the documents I'm adding
aren't really duplicates. They're very similar, but different.
I checked out the field collapsing feature patch, applied the patch but
can't get it to build successfully. Will this patch work with a nightly
build?
Thanks!
On Fri, May 15, 2009 at 7:47 PM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
>
> Matt - you may also want to detect near duplicates at index time:
>
> http://wiki.apache.org/solr/Deduplication
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
>
>
>
> ----- Original Message ----
> > From: Matt Mitchell <go...@gmail.com>
> > To: solr-user@lucene.apache.org
> > Sent: Friday, May 15, 2009 6:52:48 PM
> > Subject: grouping response docs together
> >
> > Is there a built-in mechanism for grouping similar documents together in
> the
> > response? I'd like to make it look like there is only one document with
> > multiple "hits".
> >
> > Matt
>
>
Re: grouping response docs together
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Matt - you may also want to detect near duplicates at index time:
http://wiki.apache.org/solr/Deduplication
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: Matt Mitchell <go...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Friday, May 15, 2009 6:52:48 PM
> Subject: grouping response docs together
>
> Is there a built-in mechanism for grouping similar documents together in the
> response? I'd like to make it look like there is only one document with
> multiple "hits".
>
> Matt
Re: grouping response docs together
Posted by Rohit Gandhe <ro...@gmail.com>.
Collapse component may be of interest to you
https://issues.apache.org/jira/browse/SOLR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
On Fri, May 15, 2009 at 3:52 PM, Matt Mitchell <go...@gmail.com> wrote:
> Is there a built-in mechanism for grouping similar documents together in
> the
> response? I'd like to make it look like there is only one document with
> multiple "hits".
>
> Matt
>