You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Stefan Neufeind (JIRA)" <ji...@apache.org> on 2006/05/20 20:48:30 UTC
[jira] Updated: (NUTCH-173) PerHost Crawling Policy (
crawl.ignore.external.links )
[ http://issues.apache.org/jira/browse/NUTCH-173?page=all ]
Stefan Neufeind updated NUTCH-173:
----------------------------------
Attachment: patch08-new.patch
Here is the 08-patch, corrected to work against nightly from 2006-05-20.
Also fromHost is now only generated if really needed and nutch-default.xml is patched as well. By the way: Where should a property for "crawl" be located in the config-file? In the "fetcher"-section? In that case please somebody move it up/down or rename the property before including it in the dev-tree.
But could somebody please review it quickly? I'm not sure it's 100% correct. Still investigating on my side ...
> PerHost Crawling Policy ( crawl.ignore.external.links )
> -------------------------------------------------------
>
> Key: NUTCH-173
> URL: http://issues.apache.org/jira/browse/NUTCH-173
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Versions: 0.7.1, 0.7, 0.8-dev
> Reporter: Philippe EUGENE
> Priority: Minor
> Attachments: patch.txt, patch08-new.patch, patch08.txt
>
> There is two major way of crawl in Nutch.
> Intranet Crawl : forbidden all, allow somes few host
> Whole-web crawl : allow all, forbidden few thinks
> I propose a third type of crawl.
> Directory Crawl : The purpose of this crawl is to manage few thousands of host wihtout managing rules pattern in UrlFilterRegexp.
> I made two patch for : 0.7, 0.7.1 and 0.8-dev
> I propose a new boolean property in nutch-site.xml : crawl.ignore.external.links, with false value at default.
> By default this new feature don't modify the behavior of nutch crawler.
> When you setup this property to true, the crawler don't fetch external links of the host.
> So the crawl is limited to the host that you inject at the beginning at the crawl.
> I know there is some proposal of new crawl policy using the CrawlDatum in 0.8-dev branch.
> This feature colud be a easiest way to add quickly new crawl feature to nutch, waiting for a best way to improve crawl policy.
> I post two patch.
> Sorry for my very poor english
> --
> Philippe
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira