Problem with DB_GONE status

[Andrzej Bialecki <ab...@getopt.org>]

Hi,

I have some doubts about how the current code interprets this state.

Currently, if a page in CrawlDB ends up in STATUS_DB_GONE, it will stay 
forever in that state. But it's easy to imagine a scenario where a page 
is not accessible for a while (e.g. the server is down for a longer 
period and then is restored, or someone made a mistake and linked to a 
/media/year2006/April), but then after a while it becomes available again.

As it is now, Generator will always skip pages with this status, so they 
have no chance of being ever revisited. I propose to never treat such 
pages as truly gone, but instead to increase significantly their 
re-fetch interval. This way eventually we will be able to check if the 
page is back again - if not, we increase the interval again, but if it's 
back we have a chance to reset the status to DB_FETCHED.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

Re: Problem with DB_GONE status

[Doug Cutting <cu...@apache.org>]

Andrzej Bialecki wrote:
> Currently, if a page in CrawlDB ends up in STATUS_DB_GONE, it will stay 
> forever in that state. But it's easy to imagine a scenario where a page 
> is not accessible for a while (e.g. the server is down for a longer 
> period and then is restored, or someone made a mistake and linked to a 
> /media/year2006/April), but then after a while it becomes available again.
> 
> As it is now, Generator will always skip pages with this status, so they 
> have no chance of being ever revisited. I propose to never treat such 
> pages as truly gone, but instead to increase significantly their 
> re-fetch interval. This way eventually we will be able to check if the 
> page is back again - if not, we increase the interval again, but if it's 
> back we have a chance to reset the status to DB_FETCHED.

That sounds reasonable to me.  +1

Doug

Re: Problem with DB_GONE status

[Matt Kangas <ka...@gmail.com>]

+1

I see these issues on a daily basis with my directed crawl. Sometimes  
it's due to a DNS hiccup, sometimes a route is down, sometimes it's  
me upgrading our router when the crawl is scheduled.

What's most frustrating are pages that fetch perfectly every week for  
months on end, and then are suddenly set to STATUS_DB_GONE due to a  
transient failure. I find myself wishing that the webdb had some  
memory of past success, but I realize that this would be pretty  
expensive to implement in general.

I work around this at present by periodically resetting the crawl- 
time on pages with status STATUS_DB_GONE, but that is a hideous hack.  
Andrzej's suggestion seems like an elegant solution to this problem.

--matt

On Feb 23, 2006, at 8:23 AM, Andrzej Bialecki wrote:

> Hi,
>
> I have some doubts about how the current code interprets this state.
>
> Currently, if a page in CrawlDB ends up in STATUS_DB_GONE, it will  
> stay forever in that state. But it's easy to imagine a scenario  
> where a page is not accessible for a while (e.g. the server is down  
> for a longer period and then is restored, or someone made a mistake  
> and linked to a /media/year2006/April), but then after a while it  
> becomes available again.
>
> As it is now, Generator will always skip pages with this status, so  
> they have no chance of being ever revisited. I propose to never  
> treat such pages as truly gone, but instead to increase  
> significantly their re-fetch interval. This way eventually we will  
> be able to check if the page is back again - if not, we increase  
> the interval again, but if it's back we have a chance to reset the  
> status to DB_FETCHED.
>
> -- 
> Best regards,
> Andrzej Bialecki     <><
>  ___. ___ ___ ___ _ _   __________________________________
> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
> ___|||__||  \|  ||  |  Embedded Unix, System Integration
> http://www.sigram.com  Contact: info at sigram dot com
>

--
Matt Kangas / kangas@gmail.com