Dedup - works on single file

["K.A.Hussain Ali" <Hu...@photoninfotech.com>]

HI all,

Do delete dupliates (dedup) works on single segment ?

Or do Nutch provides  any ways to delete dupliates  (dedup) on group of segments ?

I tried give the entire segments folder but it works on single segment only..

Any Help is greatly appreciated

Thanks in advance

regards
-Hussain.
 

Re: Dedup - works on single file

[Andrzej Bialecki <ab...@getopt.org>]

K.A.Hussain Ali wrote:

>HI all,
>
>Do delete dupliates (dedup) works on single segment ?
>
>  
>

Dedup works on multiple indexes. Please see the source of Crawl.main() 
for example of its use.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com