RSS based crawl - how to crawl ref links in next round

[Sourajit Basak <so...@gmail.com>]

We are crawling RSS links using a custom plugin. Thats working fine.

Our intention is to crawl the discovered urls in the subsequent round.
However, we notice that the links discovered have a status fetch_success &
also has a signature. Hence the generate phase in the subsequent round
isn't producing any urls to fetch.

We are setting a non-null empty string as parseText in the custom plugin.

Any ideas on how to force the second round ?

~ Sourajit

RE: RSS based crawl - how to crawl ref links in next round

[Markus Jelsma <ma...@openindex.io>]

 
 
-----Original message-----
> From:Sourajit Basak <so...@gmail.com>
> Sent: Tue 11-Jun-2013 14:50
> To: user@nutch.apache.org
> Subject: RSS based crawl - how to crawl ref links in next round
> 
> We are crawling RSS links using a custom plugin. Thats working fine.
> 
> Our intention is to crawl the discovered urls in the subsequent round.
> However, we notice that the links discovered have a status fetch_success &
> also has a signature.

This should not be true for NEW discovered outlinks. Parser plugins return a List<Outlink> and do not carry signature or fetch status information at all. Are you sure you haven't already crawled them?

> Hence the generate phase in the subsequent round
> isn't producing any urls to fetch.
> 
> We are setting a non-null empty string as parseText in the custom plugin.
> 
> Any ideas on how to force the second round ?
> 
> ~ Sourajit
>