You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by cao yuzhong <ca...@hotmail.com> on 2005/04/13 12:56:07 UTC
Why Crawl failed to fetch so many pages?
I want crawl all the pages in http://news.buaa.edu.cn
Following is my crawl-urlfilter.txt:
# skip file:, ftp:, & mailto: urls
-^(file|ftp|mailto|https):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|rtf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe)$
# skip URLs containing certain characters as probable queries, etc.
# -[?*!@=]
# accept anything else
#+.
+^http://news.buaa.edu.cn
I started crawl at the url: http://news.buaa.edu.cn and the crawl depth is
50.
But I found many pages in http://news.buaa.edu.cn have not been fetched.
For example:
Url http://news.buaa.edu.cn/sortnews.php?type=1 have been fetched.
It links to many pages.
Some of those pages have been fetched also.
such as
http://news.buaa.edu.cn/dispnews.php?type=1&nid=2508&s_table=news_txt
but many failed to be fetched,such as:
http://news.buaa.edu.cn/dispnews.php?type=1&nid=2500&s_table=news_txt
http://news.buaa.edu.cn/dispnews.php?type=1&nid=2504&s_table=news_txt
http://news.buaa.edu.cn/dispnews.php?type=1&nid=2505&s_table=news_txt
What's wrong about the Crawl?
Best regards.
Cao Yuzhong
2005-04-13
Re: Why Crawl failed to fetch so many pages?
Posted by Nutch开发邮件 <pr...@gmail.com>.
please modify below
（# skip URLs containing certain characters as probable queries, etc.
# -[?*!@=]）
because the link
http://news.buaa.edu.cn/dispnews.php?type=1&nid=2500&s_table=news_txt
includes the ?=& which will be ignored
it will be
(# skip URLs containing certain characters as probable queries, etc.
# -[@])
2005/4/14, Andy Liu <an...@gmail.com>:
>
> By default, Nutch only crawls the first 100 outlinks on a page. Maybe
> that's your problem?
>
> On 4/14/05, Matthias Jaekle <ja...@eventax.de> wrote:
> > > try
> > > +^http://news.buaa.edu.cn/*
> > This should not be the reason.
> > Your regex fits on urls starting with:
> > http://news.buaa.edu.cn
> > http://news.buaa.edu.cn/
> > http://news.buaa.edu.cn//
> > http://news.buaa.edu.cn/// ...
> >
> > The only thing I would try is to escape some caracters to make it more
> > correct. A dot means every possible sign. Better:
> > +^http:\/\/news\.buaa\.edu\.cn
> >
> > Did you make enough rounds, to get the wanted depth?
> > With every crawl you only fetch the already known links.
> >
> > Matthias
> >
> > --
> > http://www.eventax.com - eventax GmbH
> > http://www.umkreisfinder.de - Die Suchmaschine für Lokales und Events
> >
>
--
TEL 0512-68251233-6966
MSN:prettysino@hotmail.com
Mail:jimijinzhang@BenQ.com
QQ:58624951
BenQ.com <http://BenQ.com>
268 Shishan Road, New District,
Suzhou, China
Re: Why Crawl failed to fetch so many pages?
Posted by Andy Liu <an...@gmail.com>.
By default, Nutch only crawls the first 100 outlinks on a page. Maybe
that's your problem?
On 4/14/05, Matthias Jaekle <ja...@eventax.de> wrote:
> > try
> > +^http://news.buaa.edu.cn/*
> This should not be the reason.
> Your regex fits on urls starting with:
> http://news.buaa.edu.cn
> http://news.buaa.edu.cn/
> http://news.buaa.edu.cn//
> http://news.buaa.edu.cn/// ...
>
> The only thing I would try is to escape some caracters to make it more
> correct. A dot means every possible sign. Better:
> +^http:\/\/news\.buaa\.edu\.cn
>
> Did you make enough rounds, to get the wanted depth?
> With every crawl you only fetch the already known links.
>
> Matthias
>
> --
> http://www.eventax.com - eventax GmbH
> http://www.umkreisfinder.de - Die Suchmaschine für Lokales und Events
>
Re: Why Crawl failed to fetch so many pages?
Posted by Matthias Jaekle <ja...@eventax.de>.
> try
> +^http://news.buaa.edu.cn/*
This should not be the reason.
Your regex fits on urls starting with:
http://news.buaa.edu.cn
http://news.buaa.edu.cn/
http://news.buaa.edu.cn//
http://news.buaa.edu.cn/// ...
The only thing I would try is to escape some caracters to make it more
correct. A dot means every possible sign. Better:
+^http:\/\/news\.buaa\.edu\.cn
Did you make enough rounds, to get the wanted depth?
With every crawl you only fetch the already known links.
Matthias
--
http://www.eventax.com - eventax GmbH
http://www.umkreisfinder.de - Die Suchmaschine für Lokales und Events
Re: Why Crawl failed to fetch so many pages?
Posted by Jack Tang <hi...@gmail.com>.
try
+^http://news.buaa.edu.cn/*
On 4/13/05, cao yuzhong <ca...@hotmail.com> wrote:
>
> I want crawl all the pages in http://news.buaa.edu.cn
>
> Following is my crawl-urlfilter.txt:
>
> # skip file:, ftp:, & mailto: urls
> -^(file|ftp|mailto|https):
>
> # skip image and other suffixes we can't yet parse
> -\.(gif|GIF|jpg|JPG|ico|ICO|css|sit|eps|wmf|rtf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe)$
>
> # skip URLs containing certain characters as probable queries, etc.
> # -[?*!@=]
>
> # accept anything else
> #+.
> +^http://news.buaa.edu.cn
> I started crawl at the url: http://news.buaa.edu.cn and the crawl depth is
> 50.
> But I found many pages in http://news.buaa.edu.cn have not been fetched.
>
> For example:
> Url http://news.buaa.edu.cn/sortnews.php?type=1 have been fetched.
> It links to many pages.
> Some of those pages have been fetched also.
> such as
> http://news.buaa.edu.cn/dispnews.php?type=1&nid=2508&s_table=news_txt
>
> but many failed to be fetched,such as:
> http://news.buaa.edu.cn/dispnews.php?type=1&nid=2500&s_table=news_txt
> http://news.buaa.edu.cn/dispnews.php?type=1&nid=2504&s_table=news_txt
> http://news.buaa.edu.cn/dispnews.php?type=1&nid=2505&s_table=news_txt
>
> What's wrong about the Crawl?
>
> Best regards.
> Cao Yuzhong
> 2005-04-13
>
>