how do fetch wait times work?

[Fred Zimmerman <wf...@gmail.com>]

When I run bin/crawl once and it generates a segment list with a bunch of
fetch dates in the future, does nutch proactively run those fetches on
those future dates, or do I have to do something to make that happen?

Re: how do fetch wait times work?

[Sebastian Nagel <wa...@googlemail.com>]

Hi Fred,

Nutch does nothing "proactively", the crawl jobs must be explicitly called.
But you need no special command:
- let's say the you didn't change the defaults and
  db.fetch.interval.default == 30 days
- if you launch bin/crawl one month later, all pages are refetched,
  and optionally reindexed (404s removed)
- just to clarify: new segments will be created, old segments can be
  removed, except you need same to recover eg. if the index is lost

Best,
Sebastian

On 04/09/2018 09:13 PM, Fred Zimmerman wrote:
> When I run bin/crawl once and it generates a segment list with a bunch of
> fetch dates in the future, does nutch proactively run those fetches on
> those future dates, or do I have to do something to make that happen?
>