why are unfetchable sites kept in webdb?

[Kamil Wnuk <ka...@gmail.com>]

In UpdateDatabaseTool, the function pageGone( ... ) sets pages that have 
remained unreachable for a certain number of retries to never be fetched. Is 
there a compelling reason to keep such pages around? It seems like the right 
thing to do in this case would be to just remove the page from the webdb 
with "webdb.deletePage( oldPage )" in order to keep the webdb from 
accumulating data about pages that no longer exist. I would be happy to 
submit this change if anyone is interested, otherwise please let me know why 
the current implementation is necessary.

Thank you,
Kamil

Re: why are unfetchable sites kept in webdb?

[Doug Cutting <cu...@nutch.org>]

Kamil Wnuk wrote:
> In UpdateDatabaseTool, the function pageGone( ... ) sets pages that have 
> remained unreachable for a certain number of retries to never be fetched. Is 
> there a compelling reason to keep such pages around? It seems like the right 
> thing to do in this case would be to just remove the page from the webdb 
> with "webdb.deletePage( oldPage )" in order to keep the webdb from 
> accumulating data about pages that no longer exist. I would be happy to 
> submit this change if anyone is interested, otherwise please let me know why 
> the current implementation is necessary.

The purpose of this is that, to not waste time trying to fetch these 
pages are if other references to them are encountered.  Imagine a large 
site with a bad link on it that you re-crawl frequently.  Should we keep 
re-learning that the link is bad, or should we remember?

Doug