You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Karsten Dello <de...@mi.fu-berlin.de> on 2006/05/31 20:40:33 UTC

limited depth with internet crawl?

Hello,

I have used the intranet crawl for the following simple task:
Given a list of relevant starturls,
get all documents within the reach of two clicks.
We use this mechanism for monitoring a couple of dozen lists on the 
internet.
This was easy using the "-depth" parameter of the crawl tool.
As the number of documents was pretty small, we just recreated that 
index from scratch every two weeks.
Now the number of documents has grown,
that is why I would like to implement incremental updates.

I played around with the "whole-web"-mechanism, but I could not see how 
I can incrementally update an index while keeping the condition "max 
hops from a starturl <=2" true for all documents in the index.
I would really appreciate some advice on that.

Thanks a lot in advance,

Best regards

Karsten

Re: limited depth with internet crawl?

Posted by Andrzej Bialecki <ab...@getopt.org>.
Karsten Dello wrote:
> Hello,
>
> I have used the intranet crawl for the following simple task:
> Given a list of relevant starturls,
> get all documents within the reach of two clicks.
> We use this mechanism for monitoring a couple of dozen lists on the 
> internet.
> This was easy using the "-depth" parameter of the crawl tool.
> As the number of documents was pretty small, we just recreated that 
> index from scratch every two weeks.
> Now the number of documents has grown,
> that is why I would like to implement incremental updates.
>
> I played around with the "whole-web"-mechanism, but I could not see 
> how I can incrementally update an index while keeping the condition 
> "max hops from a starturl <=2" true for all documents in the index.
> I would really appreciate some advice on that.

In your case I think you have to store this information in 
CrawlDatum.metadata, and then implement a scoring plugin that does 
something like this (all other methods can be pass-through):

  public float generatorSortValue(UTF8 url, CrawlDatum datum, float 
initSort) throws ScoringFilterException {
    IntWritable hopCount = (IntWritable)datum.getMetaData().get(new 
UTF8("hopCount"));
    if (hopCount == null) return initSort;
    if (hopCount.get() >= 2) return 0;
    return initSort;
  }

This means that URLs with hopCount >= 2 won't ever be selected for fetching.

Of course you need to pass this value around, and increment it 
appropriately when you insert newly discovered URLs, but the 
ScoringFilter API should be sufficient for this - you can always pass 
around the values in some metadata, if not in CrawlDatum then in Content 
or ParseData.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com