You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Karsten Dello <de...@mi.fu-berlin.de> on 2006/05/31 20:40:33 UTC
limited depth with internet crawl?
Hello,
I have used the intranet crawl for the following simple task:
Given a list of relevant starturls,
get all documents within the reach of two clicks.
We use this mechanism for monitoring a couple of dozen lists on the
internet.
This was easy using the "-depth" parameter of the crawl tool.
As the number of documents was pretty small, we just recreated that
index from scratch every two weeks.
Now the number of documents has grown,
that is why I would like to implement incremental updates.
I played around with the "whole-web"-mechanism, but I could not see how
I can incrementally update an index while keeping the condition "max
hops from a starturl <=2" true for all documents in the index.
I would really appreciate some advice on that.
Thanks a lot in advance,
Best regards
Karsten
Re: limited depth with internet crawl?
Posted by Andrzej Bialecki <ab...@getopt.org>.
Karsten Dello wrote:
> Hello,
>
> I have used the intranet crawl for the following simple task:
> Given a list of relevant starturls,
> get all documents within the reach of two clicks.
> We use this mechanism for monitoring a couple of dozen lists on the
> internet.
> This was easy using the "-depth" parameter of the crawl tool.
> As the number of documents was pretty small, we just recreated that
> index from scratch every two weeks.
> Now the number of documents has grown,
> that is why I would like to implement incremental updates.
>
> I played around with the "whole-web"-mechanism, but I could not see
> how I can incrementally update an index while keeping the condition
> "max hops from a starturl <=2" true for all documents in the index.
> I would really appreciate some advice on that.
In your case I think you have to store this information in
CrawlDatum.metadata, and then implement a scoring plugin that does
something like this (all other methods can be pass-through):
public float generatorSortValue(UTF8 url, CrawlDatum datum, float
initSort) throws ScoringFilterException {
IntWritable hopCount = (IntWritable)datum.getMetaData().get(new
UTF8("hopCount"));
if (hopCount == null) return initSort;
if (hopCount.get() >= 2) return 0;
return initSort;
}
This means that URLs with hopCount >= 2 won't ever be selected for fetching.
Of course you need to pass this value around, and increment it
appropriately when you insert newly discovered URLs, but the
ScoringFilter API should be sufficient for this - you can always pass
around the values in some metadata, if not in CrawlDatum then in Content
or ParseData.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com