Crawling

[vivek <vi...@gmail.com>]

Hi,

I am Vivek and i am working on nutch.I have a few doubts regarding Crawling:

1)when a page is fetched and after 30 days  it is due for
fetching(default),but what happens if a page does not exist now on the
web.Is it removed from the crawldb and segments or it still remains  there.
In case the page is stale and I want to remove it from my crawled data how
can I do that??


2)How to refresh a crawl? .I mean suppose I have crawled 100000 urls and
after 5th depth I want that the fetching should be done again from the
beginning without stopping the process i.e continous crawl from depth 1 to
5 continuously without stopping and this too in a cyclic process


-- 







Thanks and Regards,

VIVEK KOUL