You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "kevin.Y" <02...@163.com> on 2007/08/24 12:51:52 UTC
why did nutch miss so many links when crawling?
hi!
I got a problem when using nutch-0.9 to crawl a site.
There was an article-list-page , and in this page there were many links
which point to the article-pages.
So i made nutch crawl starting with the list-page so that those articles
could be indexed.
However during the crawling i found nutch ignored all those article links !
At last , none of those articles but some other pages could be indexed. I
tried several times and got the same problem.
I'm sure there's no problem with the conf/crawl-urlfilter.txt.(
+^http://([a-z0-9]*\.)*site.com/ )
Doesn't nutch pull out all the links from a page and crawl them all? Have i
made some stupid mistakes?
any help ??
any reply will be great appreciated!
--
View this message in context: http://www.nabble.com/why-did-nutch-miss-so-many-links-when-crawling--tf4322916.html#a12310200
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: protocol not found for url=file
Posted by "kevin.Y" <02...@163.com>.
Hi MOHIT!
I haven't any experience on file-protocol crawl.But i think you should check
out the plugin.includes property in the nutch-default.xml . Are you sure
you have enabled the protocol-file plugin ?
Regards ,
Keven
MOHIT GOYAL wrote:
>
>
>
> --
> I tried to crawl the local directory files by giving links to local
> directory in urls.I got the following error.
>
> command:
> bin/nutch crawl ../urls -dir crawlresult_localfs1
>
>
> please help
>
> -------------------------------------------------
> failed with: org.apache.nutch.protocol.ProtocolNotFound: protocol not
> found for url=file
> fetching file:///root/Desktop/csiro-split/CSIRO002
>
>
>
>
>
>
>
> MOHIT GOYAL
> CSE
> 200502013
>
>
>
>
>
>
>
--
View this message in context: http://www.nabble.com/why-did-nutch-miss-so-many-links-when-crawling--tf4322916.html#a12322096
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: protocol not found for url=file
Posted by MOHIT GOYAL <go...@students.iiit.ac.in>.
--
I tried to crawl the local directory files by giving links to local
directory in urls.I got the following error.
command:
bin/nutch crawl ../urls -dir crawlresult_localfs1
please help
-------------------------------------------------
failed with: org.apache.nutch.protocol.ProtocolNotFound: protocol not
found for url=file
fetching file:///root/Desktop/csiro-split/CSIRO002
MOHIT GOYAL
CSE
200502013
RE: why did nutch miss so many links when crawling?
Posted by "kevin.Y" <02...@163.com>.
Thank you, Vishal !
I modified the db.max.outlinks.per.page property as you said. I think that
is the problem. And it works ! Thanks a lot !
Regards,
Keven
Vishal Shah-3 wrote:
>
> Hi Kevin,
>
> Are you using the crawl tool for crawling? If no, then you should look
> at
> the regex patterns in regex-urlfilter.txt.
>
> Also, look at the db.max.outlinks.per.page property in nutch-site.xml.
> This property determines how many outlinks from a page are processed.
>
> Regards,
>
> -vishal.
>
> -----Original Message-----
> From: kevin.Y [mailto:027hk@163.com]
> Sent: Friday, August 24, 2007 4:22 PM
> To: nutch-user@lucene.apache.org
> Subject: why did nutch miss so many links when crawling?
>
>
> hi!
> I got a problem when using nutch-0.9 to crawl a site.
> There was an article-list-page , and in this page there were many links
> which point to the article-pages.
> So i made nutch crawl starting with the list-page so that those articles
> could be indexed.
> However during the crawling i found nutch ignored all those article links
> !
> At last , none of those articles but some other pages could be indexed. I
> tried several times and got the same problem.
> I'm sure there's no problem with the conf/crawl-urlfilter.txt.(
> +^http://([a-z0-9]*\.)*site.com/ )
> Doesn't nutch pull out all the links from a page and crawl them all? Have
> i
> made some stupid mistakes?
> any help ??
>
> any reply will be great appreciated!
> --
> View this message in context:
> http://www.nabble.com/why-did-nutch-miss-so-many-links-when-crawling--tf4322
> 916.html#a12310200
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context: http://www.nabble.com/why-did-nutch-miss-so-many-links-when-crawling--tf4322916.html#a12322016
Sent from the Nutch - User mailing list archive at Nabble.com.
RE: why did nutch miss so many links when crawling?
Posted by Vishal Shah <vi...@rediff.co.in>.
Hi Kevin,
Are you using the crawl tool for crawling? If no, then you should look at
the regex patterns in regex-urlfilter.txt.
Also, look at the db.max.outlinks.per.page property in nutch-site.xml.
This property determines how many outlinks from a page are processed.
Regards,
-vishal.
-----Original Message-----
From: kevin.Y [mailto:027hk@163.com]
Sent: Friday, August 24, 2007 4:22 PM
To: nutch-user@lucene.apache.org
Subject: why did nutch miss so many links when crawling?
hi!
I got a problem when using nutch-0.9 to crawl a site.
There was an article-list-page , and in this page there were many links
which point to the article-pages.
So i made nutch crawl starting with the list-page so that those articles
could be indexed.
However during the crawling i found nutch ignored all those article links !
At last , none of those articles but some other pages could be indexed. I
tried several times and got the same problem.
I'm sure there's no problem with the conf/crawl-urlfilter.txt.(
+^http://([a-z0-9]*\.)*site.com/ )
Doesn't nutch pull out all the links from a page and crawl them all? Have i
made some stupid mistakes?
any help ??
any reply will be great appreciated!
--
View this message in context:
http://www.nabble.com/why-did-nutch-miss-so-many-links-when-crawling--tf4322
916.html#a12310200
Sent from the Nutch - User mailing list archive at Nabble.com.