You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Joshua J Pavel <jp...@us.ibm.com> on 2010/05/18 01:31:34 UTC

Fetch Interval and AddDays

I would like to recrawl a certain site I admin at a much smaller interval 
- say, every hour.

I've specified my db.fetch.interval.default to be 1200, but if I attempt 
to recrawl using the crawl directory and segments from last time, I can't 
seem to get it to work.  I think the problem is adddays... is there any 
way to reuse my crawl segments with a fetch segment less than 1 day?  I 
currently delete my old crawl directory.  While that works, I would like 
to preserve the entries and timestamps and such in there between crawls. 
Is this possible?

Re: Fetch Interval and AddDays

Posted by Davide Cavalaglio <da...@desktopsrl.com>.
Hi,
I have the same question of Joshua. I not understand how do a recrawl on
existing crawldb without delete the crawldb.
Someone can explain me how do this tasks? I'm newbye with Nutch.

Thank and sorry from my poor english.

2010/5/18 Joshua J Pavel <jp...@us.ibm.com>

> I would like to recrawl a certain site I admin at a much smaller interval
> - say, every hour.
>
> I've specified my db.fetch.interval.default to be 1200, but if I attempt
> to recrawl using the crawl directory and segments from last time, I can't
> seem to get it to work.  I think the problem is adddays... is there any
> way to reuse my crawl segments with a fetch segment less than 1 day?  I
> currently delete my old crawl directory.  While that works, I would like
> to preserve the entries and timestamps and such in there between crawls.
> Is this possible?