Dedup remove all duplicates

[Pascal Dimassimo <th...@hotmail.com>]

Hi,

After I run the index command, my index contains 2 documents with the same
boost, digest, segment and title, but with different tstamp and url. When I
run the dedup command on that index, both documents are removed.

Should the document with the latest tstamp be kept?

Thanks!
-- 
View this message in context: http://old.nabble.com/Dedup-remove-all-duplicates-tp27026925p27026925.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Dedup remove all duplicates

[Andrzej Bialecki <ab...@getopt.org>]

On 2010-01-06 18:56, Pascal Dimassimo wrote:
>
> Hi,
>
> After I run the index command, my index contains 2 documents with the same
> boost, digest, segment and title, but with different tstamp and url. When I
> run the dedup command on that index, both documents are removed.
>
> Should the document with the latest tstamp be kept?

It should, out of multiple documents with the same URL (url duplicates) 
only the most recent is retained - unless it was removed because there 
was another document in the index with the same content (a content 
duplicate).

Could you please verify this on a minimal index (2 documents), and if 
the problem persist please report this in JIRA.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com