You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by zud <pr...@gmail.com> on 2010/01/23 09:31:43 UTC
Crawl depth problem
hi
i am running nutch crawl and i have specified the depth as 200 but in the
console it is showing Stopping at depth=1 - no more URLs to fetch.
it is not crawling the website completely
even i cant find the errors in log file
please help me out
--
View this message in context: http://old.nabble.com/Crawl-depth-problem-tp27284306p27284306.html
Sent from the Nutch - User mailing list archive at Nabble.com.
Re: Crawl depth problem
Posted by MilleBii <mi...@gmail.com>.
If you use crawl command you need to set crawl-filters.txt for your
site (not the regex filter)
Depth of 200 is huge, becareful depth actually means the number of
fetch/parse/index cycles.
2010/1/23, Lyndon Maydwell <ma...@gmail.com>:
> Have you set up your regex-urlfilter.txt correctly? I've been caught
> out by this before.
>
> On Sat, Jan 23, 2010 at 4:31 PM, zud <pr...@gmail.com> wrote:
>>
>> hi
>>
>> i am running nutch crawl and i have specified the depth as 200 but in
>> the
>> console it is showing Stopping at depth=1 - no more URLs to fetch.
>>
>> it is not crawling the website completely
>> even i cant find the errors in log file
>>
>> please help me out
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Crawl-depth-problem-tp27284306p27284306.html
>> Sent from the Nutch - User mailing list archive at Nabble.com.
>>
>>
>
--
-MilleBii-
Re: Crawl depth problem
Posted by Lyndon Maydwell <ma...@gmail.com>.
Have you set up your regex-urlfilter.txt correctly? I've been caught
out by this before.
On Sat, Jan 23, 2010 at 4:31 PM, zud <pr...@gmail.com> wrote:
>
> hi
>
> i am running nutch crawl and i have specified the depth as 200 but in the
> console it is showing Stopping at depth=1 - no more URLs to fetch.
>
> it is not crawling the website completely
> even i cant find the errors in log file
>
> please help me out
>
> --
> View this message in context: http://old.nabble.com/Crawl-depth-problem-tp27284306p27284306.html
> Sent from the Nutch - User mailing list archive at Nabble.com.
>
>