You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Shane Wood <sh...@cbm8bit.com> on 2015/04/22 04:19:45 UTC

Help. Nutch not crawling site.

Is there any reason why Nutch would not crawl this site ? 
http://www.bitfixer.com

I have checked and can not find a robots.txt file.

Cheers
Shane.

Re: Help. Nutch not crawling site.

Posted by Michael Joyce <jo...@apache.org>.

What is your config? What happens when you run the crawl? What is in your
crawldb? What are your seed urls? What is the output of parsechecker on
that sight? If you can throw some info our way I'm sure we can help you
figure out why you're having problems.

-- Jimmy

On Tue, Apr 21, 2015 at 7:19 PM, Shane Wood <sh...@cbm8bit.com> wrote:

> Is there any reason why Nutch would not crawl this site ?
> http://www.bitfixer.com
>
> I have checked and can not find a robots.txt file.
>
> Cheers
> Shane.
>