You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by "Håvard W. Kongsgård" <h....@niap.no> on 2005/11/22 16:38:47 UTC

Intranet craw folder

Hi, I am still testing nutch 0.7.1 but now I have another problem.
When I do a normal intranet crawl on some web folders with 2000 pdfs, 
nutch only fetches 47 pdfs from each folder.

Re: Intranet craw folder

Posted by "Håvard W. Kongsgård" <h....@niap.no>.

Do you mean http.content.limit? I have set it to -1 already.

There are no "Content truncated at 65536 bytes. Parser can't handle 
incomplete" errors in the log.

Stefan Groschupf wrote:

> Check the maximal content limit in nutch-default.xml
>
> Am 22.11.2005 um 16:38 schrieb Håvard W. Kongsgård:
>
>> Hi, I am still testing nutch 0.7.1 but now I have another problem.
>> When I do a normal intranet crawl on some web folders with 2000  
>> pdfs, nutch only fetches 47 pdfs from each folder.
>>
>>
>
>
>
>

Re: Intranet craw folder

Posted by Stefan Groschupf <sg...@media-style.com>.

Check the maximal content limit in nutch-default.xml

Am 22.11.2005 um 16:38 schrieb Håvard W. Kongsgård:

> Hi, I am still testing nutch 0.7.1 but now I have another problem.
> When I do a normal intranet crawl on some web folders with 2000  
> pdfs, nutch only fetches 47 pdfs from each folder.
>
>