You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Mark <st...@gmail.com> on 2011/03/05 07:49:28 UTC

Detecting duplicates

Is there a way one could detect duplicates (say by using some unique 
hash of certain fields) and marking a document as a duplicate but not 
remove it.

Here is an example:

Doc 1) This is my test
Doc 2) This is my test
Doc 3) Another test
Doc 4) This is my test

Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked 
as duplicates (of doc 1).

Can this be easily accomplished?

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Detecting duplicates

Posted by Li Li <fa...@gmail.com>.
it's the problem of near duplication detection. there are many papers
addressing this problem. methods like simhash are used.

2011/3/5 Mark <st...@gmail.com>

> Is there a way one could detect duplicates (say by using some unique hash
> of certain fields) and marking a document as a duplicate but not remove it.
>
> Here is an example:
>
> Doc 1) This is my test
> Doc 2) This is my test
> Doc 3) Another test
> Doc 4) This is my test
>
> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked as
> duplicates (of doc 1).
>
> Can this be easily accomplished?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Detecting duplicates

Posted by "Devon H. O'Dell" <de...@gmail.com>.
There is a DuplicateFilter class in contrib that works pretty well.

2011/3/5 Grant Ingersoll <gs...@apache.org>:
> See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy to pull out if you are doing just Lucene.
>
> On Mar 5, 2011, at 1:49 AM, Mark wrote:
>
>> Is there a way one could detect duplicates (say by using some unique hash of certain fields) and marking a document as a duplicate but not remove it.
>>
>> Here is an example:
>>
>> Doc 1) This is my test
>> Doc 2) This is my test
>> Doc 3) Another test
>> Doc 4) This is my test
>>
>> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked as duplicates (of doc 1).
>>
>> Can this be easily accomplished?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem docs using Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Detecting duplicates

Posted by Otis Gospodnetic <ot...@yahoo.com>.
Mark,

Keep in mind that there are actually multiple patches for this.  SOLR-236 and 
SOLR-1086 IIRC.
Also, I just noticed this is java-user@lucene.  You may want to continue on 
solr-user@lucene.

Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



----- Original Message ----
> From: Mark <st...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Sat, March 5, 2011 8:35:13 PM
> Subject: Re: Detecting duplicates
> 
> I'm familiar with Deduplication however I do not wish to remove my 
> duplicates and my needs are slightly different. I would like to mark the 
> first document with signature 'xyz' as unique but the next one as a 
> duplicate. This way I can filter out "duplicates" during searching using 
> a filter query but still return the original document.
> 
> The only thing  I know of at the moment is to use field collapsing but I 
> tried the patch on  1.4.1 and it was terribly slow.
> 
> On 3/5/11 4:43 AM, Grant Ingersoll  wrote:
> > See http://wiki.apache.org/solr/Deduplication.  Should be  fairly easy to 
>pull out if you are doing just Lucene.
> >
> > On Mar 5,  2011, at 1:49 AM, Mark wrote:
> >
> >> Is there a way one could  detect duplicates (say by using some unique hash 
>of certain fields) and marking  a document as a duplicate but not remove it.
> >>
> >> Here is an  example:
> >>
> >> Doc 1) This is my test
> >> Doc 2) This  is my test
> >> Doc 3) Another test
> >> Doc 4) This is my  test
> >>
> >> Doc 1 and 3 should be considered unique whereas 2  and 4 should be marked as 
>duplicates (of doc 1).
> >>
> >> Can  this be easily accomplished?
> >>
> >>  ---------------------------------------------------------------------
> >>  To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>  For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >  --------------------------
> > Grant Ingersoll
> > http://www.lucidimagination.com/
> >
> > Search the Lucene  ecosystem docs using Solr/Lucene:
> > http://www.lucidimagination.com/search
> >
> >
> >  ---------------------------------------------------------------------
> > To  unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >  For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> 
> ---------------------------------------------------------------------
> To  unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For  additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Detecting duplicates

Posted by Alexander Aristov <al...@gmail.com>.
did you check it
http://wiki.apache.org/solr/Deduplication


Best Regards
Alexander Aristov


On 10 March 2011 18:35, Mark <st...@gmail.com> wrote:

> My understanding is It can mark documents with the same signature
> indicating that they are similar however there is no way at query time to
> return only 1 "unique" document per signature. Am I missing something?
>
>
> Doc 1) This is my test
> Doc 2) This is my test
> Doc 3) Another test
> Doc 4) This is my test
>
> If I run a query for "test" it should return
>
>
> Doc 1) This is my test
> Doc 3) Another test
>
>
>
> On 3/10/11 6:25 AM, Grant Ingersoll wrote:
>
>> On Mar 5, 2011, at 8:35 PM, Mark wrote:
>>
>>  I'm familiar with Deduplication however I do not wish to remove my
>>> duplicates and my needs are slightly different. I would like to mark the
>>> first document with signature 'xyz' as unique but the next one as a
>>> duplicate. This way I can filter out "duplicates" during searching using a
>>> filter query but still return the original document.
>>>
>> My understanding is that you can have it mark duplicates.
>>
>>  The only thing I know of at the moment is to use field collapsing but I
>>> tried the patch on 1.4.1 and it was terribly slow.
>>>
>>> On 3/5/11 4:43 AM, Grant Ingersoll wrote:
>>>
>>>> See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy
>>>> to pull out if you are doing just Lucene.
>>>>
>>>> On Mar 5, 2011, at 1:49 AM, Mark wrote:
>>>>
>>>>  Is there a way one could detect duplicates (say by using some unique
>>>>> hash of certain fields) and marking a document as a duplicate but not remove
>>>>> it.
>>>>>
>>>>> Here is an example:
>>>>>
>>>>> Doc 1) This is my test
>>>>> Doc 2) This is my test
>>>>> Doc 3) Another test
>>>>> Doc 4) This is my test
>>>>>
>>>>> Doc 1 and 3 should be considered unique whereas 2 and 4 should be
>>>>> marked as duplicates (of doc 1).
>>>>>
>>>>> Can this be easily accomplished?
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>
>>>>>  --------------------------
>>>> Grant Ingersoll
>>>> http://www.lucidimagination.com/
>>>>
>>>> Search the Lucene ecosystem docs using Solr/Lucene:
>>>> http://www.lucidimagination.com/search
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>>>  ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>  --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem docs using Solr/Lucene:
>> http://www.lucidimagination.com/search
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Detecting duplicates

Posted by mark harwood <ma...@yahoo.co.uk>.
This is possible using contrib's DuplicateFilter.
Below is an example of your problem defined as an XML-based test which I just 
ran OK through my test writer/runner.
Hopefully this is readable and demonstrates the use of 
FilteredQuery/DuplicateFilter.


<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="test.xsl"?>
<Test description="DuplicateFilter tests">
 <Data>
  <Index name="index1">
   <Analyzers 
class="org.apache.lucene.analysis.standard.StandardAnalyzer29"> </Analyzers>
   <Shard name="shard1">
    <Document pk="1">
     <Field name="text">This is my test</Field>
     <Field name="md5">abc</Field>
    </Document>
    <Document pk="2">
     <Field name="text">This is my test</Field>
     <Field name="md5">abc</Field>
    </Document>
    <Document pk="3">
     <Field name="text">Another test</Field>
     <Field name="md5">def</Field>
    </Document>
    <Document pk="4">
     <Field name="text">This is my test</Field>
     <Field name="md5">abc</Field>
    </Document>
   </Shard>
  </Index>
 </Data>
 <Tests>
  <Test description="Eliminate duplicates based on MD5 field">
   <Query>
    <FilteredQuery>
     <Query>
      <UserQuery fieldName="text">test</UserQuery>
     </Query>
     <Filter>
      <DuplicateFilter fieldName="md5"/>
     </Filter>
    </FilteredQuery>
   </Query>
   <ExpectedResults>
    <Result fieldName="pk">1</Result>
    <Result fieldName="pk">3</Result>
   </ExpectedResults>
  </Test>  
 </Tests>
</Test>


----- Original Message ----
From: Mark <st...@gmail.com>
To: java-user@lucene.apache.org
Sent: Thu, 10 March, 2011 15:35:22
Subject: Re: Detecting duplicates

My understanding is It can mark documents with the same signature 
indicating that they are similar however there is no way at query time 
to return only 1 "unique" document per signature. Am I missing something?

Doc 1) This is my test
Doc 2) This is my test
Doc 3) Another test
Doc 4) This is my test

If I run a query for "test" it should return

Doc 1) This is my test
Doc 3) Another test


On 3/10/11 6:25 AM, Grant Ingersoll wrote:
> On Mar 5, 2011, at 8:35 PM, Mark wrote:
>
>> I'm familiar with Deduplication however I do not wish to remove my duplicates 
>>and my needs are slightly different. I would like to mark the first document 
>>with signature 'xyz' as unique but the next one as a duplicate. This way I can 
>>filter out "duplicates" during searching using a filter query but still return 
>>the original document.
> My understanding is that you can have it mark duplicates.
>
>> The only thing I know of at the moment is to use field collapsing but I tried 
>>the patch on 1.4.1 and it was terribly slow.
>>
>> On 3/5/11 4:43 AM, Grant Ingersoll wrote:
>>> See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy to pull 
>>>out if you are doing just Lucene.
>>>
>>> On Mar 5, 2011, at 1:49 AM, Mark wrote:
>>>
>>>> Is there a way one could detect duplicates (say by using some unique hash of 
>>>>certain fields) and marking a document as a duplicate but not remove it.
>>>>
>>>> Here is an example:
>>>>
>>>> Doc 1) This is my test
>>>> Doc 2) This is my test
>>>> Doc 3) Another test
>>>> Doc 4) This is my test
>>>>
>>>> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked as 
>>>>duplicates (of doc 1).
>>>>
>>>> Can this be easily accomplished?
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>> --------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>>
>>> Search the Lucene ecosystem docs using Solr/Lucene:
>>> http://www.lucidimagination.com/search
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem docs using Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


      

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Detecting duplicates

Posted by Mark <st...@gmail.com>.
My understanding is It can mark documents with the same signature 
indicating that they are similar however there is no way at query time 
to return only 1 "unique" document per signature. Am I missing something?

Doc 1) This is my test
Doc 2) This is my test
Doc 3) Another test
Doc 4) This is my test

If I run a query for "test" it should return

Doc 1) This is my test
Doc 3) Another test


On 3/10/11 6:25 AM, Grant Ingersoll wrote:
> On Mar 5, 2011, at 8:35 PM, Mark wrote:
>
>> I'm familiar with Deduplication however I do not wish to remove my duplicates and my needs are slightly different. I would like to mark the first document with signature 'xyz' as unique but the next one as a duplicate. This way I can filter out "duplicates" during searching using a filter query but still return the original document.
> My understanding is that you can have it mark duplicates.
>
>> The only thing I know of at the moment is to use field collapsing but I tried the patch on 1.4.1 and it was terribly slow.
>>
>> On 3/5/11 4:43 AM, Grant Ingersoll wrote:
>>> See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy to pull out if you are doing just Lucene.
>>>
>>> On Mar 5, 2011, at 1:49 AM, Mark wrote:
>>>
>>>> Is there a way one could detect duplicates (say by using some unique hash of certain fields) and marking a document as a duplicate but not remove it.
>>>>
>>>> Here is an example:
>>>>
>>>> Doc 1) This is my test
>>>> Doc 2) This is my test
>>>> Doc 3) Another test
>>>> Doc 4) This is my test
>>>>
>>>> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked as duplicates (of doc 1).
>>>>
>>>> Can this be easily accomplished?
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>
>>> --------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com/
>>>
>>> Search the Lucene ecosystem docs using Solr/Lucene:
>>> http://www.lucidimagination.com/search
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem docs using Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Detecting duplicates

Posted by Grant Ingersoll <gs...@apache.org>.
On Mar 5, 2011, at 8:35 PM, Mark wrote:

> I'm familiar with Deduplication however I do not wish to remove my duplicates and my needs are slightly different. I would like to mark the first document with signature 'xyz' as unique but the next one as a duplicate. This way I can filter out "duplicates" during searching using a filter query but still return the original document.

My understanding is that you can have it mark duplicates.  

> 
> The only thing I know of at the moment is to use field collapsing but I tried the patch on 1.4.1 and it was terribly slow.
> 
> On 3/5/11 4:43 AM, Grant Ingersoll wrote:
>> See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy to pull out if you are doing just Lucene.
>> 
>> On Mar 5, 2011, at 1:49 AM, Mark wrote:
>> 
>>> Is there a way one could detect duplicates (say by using some unique hash of certain fields) and marking a document as a duplicate but not remove it.
>>> 
>>> Here is an example:
>>> 
>>> Doc 1) This is my test
>>> Doc 2) This is my test
>>> Doc 3) Another test
>>> Doc 4) This is my test
>>> 
>>> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked as duplicates (of doc 1).
>>> 
>>> Can this be easily accomplished?
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>> 
>> --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>> 
>> Search the Lucene ecosystem docs using Solr/Lucene:
>> http://www.lucidimagination.com/search
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem docs using Solr/Lucene:
http://www.lucidimagination.com/search


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Detecting duplicates

Posted by Li Li <fa...@gmail.com>.
it's indeed very slow. because it do collapsing in all matched documents.
we tacked this problem by doing collapsing in top 100 documents.

2011/3/6 Mark <st...@gmail.com>

> I'm familiar with Deduplication however I do not wish to remove my
> duplicates and my needs are slightly different. I would like to mark the
> first document with signature 'xyz' as unique but the next one as a
> duplicate. This way I can filter out "duplicates" during searching using a
> filter query but still return the original document.
>
> The only thing I know of at the moment is to use field collapsing but I
> tried the patch on 1.4.1 and it was terribly slow.
>
>
> On 3/5/11 4:43 AM, Grant Ingersoll wrote:
>
>> See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy to
>> pull out if you are doing just Lucene.
>>
>> On Mar 5, 2011, at 1:49 AM, Mark wrote:
>>
>>  Is there a way one could detect duplicates (say by using some unique hash
>>> of certain fields) and marking a document as a duplicate but not remove it.
>>>
>>> Here is an example:
>>>
>>> Doc 1) This is my test
>>> Doc 2) This is my test
>>> Doc 3) Another test
>>> Doc 4) This is my test
>>>
>>> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked
>>> as duplicates (of doc 1).
>>>
>>> Can this be easily accomplished?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>  --------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com/
>>
>> Search the Lucene ecosystem docs using Solr/Lucene:
>> http://www.lucidimagination.com/search
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: Detecting duplicates

Posted by Mark <st...@gmail.com>.
I'm familiar with Deduplication however I do not wish to remove my 
duplicates and my needs are slightly different. I would like to mark the 
first document with signature 'xyz' as unique but the next one as a 
duplicate. This way I can filter out "duplicates" during searching using 
a filter query but still return the original document.

The only thing I know of at the moment is to use field collapsing but I 
tried the patch on 1.4.1 and it was terribly slow.

On 3/5/11 4:43 AM, Grant Ingersoll wrote:
> See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy to pull out if you are doing just Lucene.
>
> On Mar 5, 2011, at 1:49 AM, Mark wrote:
>
>> Is there a way one could detect duplicates (say by using some unique hash of certain fields) and marking a document as a duplicate but not remove it.
>>
>> Here is an example:
>>
>> Doc 1) This is my test
>> Doc 2) This is my test
>> Doc 3) Another test
>> Doc 4) This is my test
>>
>> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked as duplicates (of doc 1).
>>
>> Can this be easily accomplished?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem docs using Solr/Lucene:
> http://www.lucidimagination.com/search
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Detecting duplicates

Posted by Grant Ingersoll <gs...@apache.org>.
See http://wiki.apache.org/solr/Deduplication.  Should be fairly easy to pull out if you are doing just Lucene.

On Mar 5, 2011, at 1:49 AM, Mark wrote:

> Is there a way one could detect duplicates (say by using some unique hash of certain fields) and marking a document as a duplicate but not remove it.
> 
> Here is an example:
> 
> Doc 1) This is my test
> Doc 2) This is my test
> Doc 3) Another test
> Doc 4) This is my test
> 
> Doc 1 and 3 should be considered unique whereas 2 and 4 should be marked as duplicates (of doc 1).
> 
> Can this be easily accomplished?
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem docs using Solr/Lucene:
http://www.lucidimagination.com/search


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org