Nutch, SolrCloud and deduplication

[Rafał Kuć <ra...@alud.com.pl>]

Hello!

We are running Nutch and SolrCloud cluster - we index our data to
SolrCloud. The problem is that we have duplicates and we would like to
get rid of those. To put it simple, we would like to remove documents
with the same content (they can have slightly different URL
addresses).

I know that there is Solr based deduplication that would work here
(http://wiki.apache.org/solr/Deduplication), but the problem is that
it doesn't work with SolrCloud.

Maybe there is another way that would allow us to remove duplicates
based on the content of the documents?

Any help would be appreciated.

-- 
Regards,
 Rafał Kuć
 Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch

RE: Nutch, SolrCloud and deduplication

[Markus Jelsma <ma...@openindex.io>]

Nutch 1.8 is close to being released and has better on board deduplication than it had before, it now works independently of indexing backend, see NUTCH-656. It will work with SolrCloud. Another method it to collapse on the doc's indexed digest but it requires sharding by digest and distributed stats to make up for the unbalanced shards, see SOLR-1632 for a working patch.

-----Original message-----
> From:Rafał Kuć <ra...@alud.com.pl>
> Sent: Wednesday 4th December 2013 15:09
> To: user@nutch.apache.org
> Subject: Nutch, SolrCloud and deduplication
> 
> Hello!
> 
> We are running Nutch and SolrCloud cluster - we index our data to
> SolrCloud. The problem is that we have duplicates and we would like to
> get rid of those. To put it simple, we would like to remove documents
> with the same content (they can have slightly different URL
> addresses).
> 
> I know that there is Solr based deduplication that would work here
> (http://wiki.apache.org/solr/Deduplication), but the problem is that
> it doesn't work with SolrCloud.
> 
> Maybe there is another way that would allow us to remove duplicates
> based on the content of the documents?
> 
> Any help would be appreciated.
> 
> -- 
> Regards,
>  Rafał Kuć
>  Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch
> 
>