[jira] Closed: (NUTCH-67) I want crawl the websites including news.yahoo.com,game.yahoo.com,blog.yahoo.com,etc!

["Stefan Groschupf (JIRA)" <ji...@apache.org>]

     [ http://issues.apache.org/jira/browse/NUTCH-67?page=all ]
     
Stefan Groschupf closed NUTCH-67:
---------------------------------

    Resolution: Won't Fix

Please ask such questions in the nutch-user mailing list.

> I  want  crawl the websites including news.yahoo.com,game.yahoo.com,blog.yahoo.com,etc!
> ---------------------------------------------------------------------------------------
>
>          Key: NUTCH-67
>          URL: http://issues.apache.org/jira/browse/NUTCH-67
>      Project: Nutch
>         Type: Wish
>   Components: fetcher
>  Environment: Windows 2000,weblogic
>     Reporter: zhangjin

>
> how do  I  config them in the crawl-urlfilter.txt? I  config  them below,but  it is not successful.
> # The url filter file used by the crawl command.
> # Better for intranet crawling.
> # Be sure to change MY.DOMAIN.NAME to your domain name.
> # Each non-comment, non-blank line contains a regular expression
> # prefixed by '+' or '-'.  The first matching pattern in the file
> # determines whether a URL is included or ignored.  If no pattern
> # matches, the URL is ignored.
> # skip file:, ftp:, & mailto: urls
> -^(file|ftp|mailto):
> # skip image and other suffixes we can't yet parse
> -\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe)$
> # skip URLs containing certain characters as probable queries, etc.
> -[?*!@=]
> # accept hosts in MY.DOMAIN.NAME
> #+^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/
> +^http://([a-z0-9]*\.)*yahoo.com/
> # skip everything else
> #-.
> but It can not work, and can not crawl the  domain name (DOMAIN.NAME) inluding  news.yahoo.com,game.yahoo.com,blog.yahoo.com
> why?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira