You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by "Harmesh, V2solutions" <ha...@in.v2solutions.com> on 2007/03/12 14:17:01 UTC

how to remove duplicate URL's

hi, 
   I have got 30,000 urls  during my crawl cycle which are not unique , so i
had to find out that how many are the unquie URL's as i know that "dedup"
will remove the duplicate urls, but i am trying to remove the duplicate
links using dedup command ,still it is not removing them can any one please
help me out . The command which i had tried is below 

   $ bin/nutch dedup <path of index dir>/indexes 

thanks in advance 

-- 
View this message in context: http://www.nabble.com/how-to-remove-duplicate-URL%27s-tf3389407.html#a9434347
Sent from the Nutch - User mailing list archive at Nabble.com.