You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by jepse <jp...@jepse.net> on 2012/04/03 12:01:01 UTC

Nutch simple doesnt crawl webpages

Hey there ;)

May anybody help me out, why nutch simply doesnt find outlinks for the
following page?
http://www.soccer-forum.de/

When i copy a link out of the site to my testsite i the Crwaler find the
added outlink. But within this page it simply doesnt find any outlinklink. 

any suggenstions?

Thanks to all!



--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-simple-doesnt-crawl-webpages-tp3880377p3880377.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Nutch simple doesnt crawl webpages

Posted by jepse <jp...@jepse.net>.
Uhhhaaaa,

i'm dealing with almost all config file. robots.txt etc... and finally it's
so easy ;)

Thanks a lot..

--
View this message in context: http://lucene.472066.n3.nabble.com/Nutch-Crawler-simple-doesnt-find-outlinks-tp3880377p3880397.html
Sent from the Nutch - User mailing list archive at Nabble.com.

Re: Nutch simple doesnt crawl webpages

Posted by Julien Nioche <li...@gmail.com>.
that page contains :

<meta name="Robots" content="INDEX,NOFOLLOW">

which I believe is self explanatory

Julien

On 3 April 2012 11:01, jepse <jp...@jepse.net> wrote:

> Hey there ;)
>
> May anybody help me out, why nutch simply doesnt find outlinks for the
> following page?
> http://www.soccer-forum.de/
>
> When i copy a link out of the site to my testsite i the Crwaler find the
> added outlink. But within this page it simply doesnt find any outlinklink.
>
> any suggenstions?
>
> Thanks to all!
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Nutch-simple-doesnt-crawl-webpages-tp3880377p3880377.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>



-- 
*
*Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble