You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2006/07/19 19:34:14 UTC
[jira] Closed: (NUTCH-173) PerHost Crawling Policy (
crawl.ignore.external.links )
[ http://issues.apache.org/jira/browse/NUTCH-173?page=all ]
Andrzej Bialecki closed NUTCH-173.
-----------------------------------
Resolution: Fixed
Patch applied to trunk/ . Thank you!
> PerHost Crawling Policy ( crawl.ignore.external.links )
> -------------------------------------------------------
>
> Key: NUTCH-173
> URL: http://issues.apache.org/jira/browse/NUTCH-173
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 0.7.1, 0.7, 0.8-dev
> Reporter: Philippe EUGENE
> Priority: Minor
> Attachments: patch.txt, patch08-new.patch, patch08.txt
>
>
> There is two major way of crawl in Nutch.
> Intranet Crawl : forbidden all, allow somes few host
> Whole-web crawl : allow all, forbidden few thinks
> I propose a third type of crawl.
> Directory Crawl : The purpose of this crawl is to manage few thousands of host wihtout managing rules pattern in UrlFilterRegexp.
> I made two patch for : 0.7, 0.7.1 and 0.8-dev
> I propose a new boolean property in nutch-site.xml : crawl.ignore.external.links, with false value at default.
> By default this new feature don't modify the behavior of nutch crawler.
> When you setup this property to true, the crawler don't fetch external links of the host.
> So the crawl is limited to the host that you inject at the beginning at the crawl.
> I know there is some proposal of new crawl policy using the CrawlDatum in 0.8-dev branch.
> This feature colud be a easiest way to add quickly new crawl feature to nutch, waiting for a best way to improve crawl policy.
> I post two patch.
> Sorry for my very poor english
> --
> Philippe
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira