You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "shubham.gupta" <sh...@orkash.com> on 2017/04/12 04:55:15 UTC

Unable to parse a huge list of seed URLs | Nutch 2.3.1 + MongoDB + Hadoop 2.7.1

Hey

I have around 5000 URLs in my seed Url list. If I inject the whole list, 
then it fails to fetch all documents and parse. The depth is set to 1.

But when the list is divided into a batch of 1000 URLs then it is able 
to fetch and parse all documents successfully.

In the former case 5141 URLs are injected, out of which 5127 URLs are 
generated and only 1300 URLs get fetched with status 2. Out of the rest 
1342 do not have a status 2 and the rest are unfetched.

While, when the list is small, the total count of documents is 3220, out 
of which the documents with status 2 are 1298, the documents with status 
code other than 2 are 1922. And, the count of documents which have not 
been fetched yet is 1.

Is this a Problem of The nutch which fails to fetch huge list of URLs? 
Or some changes need to be made in the configuration files.

Please reply soon.

-- 
Thanks and Regards,
Shubham Gupta


Re: Unable to parse a huge list of seed URLs | Nutch 2.3.1 + MongoDB + Hadoop 2.7.1

Posted by Sebastian Nagel <wa...@googlemail.com>.
> Is this a Problem of The nutch which fails to fetch huge list of URLs?

Probably not, it is able to fetch millions of URLs in a single fetch list.

Is there a timelimit set (property fetcher.timelimit.mins). That could explain
why a large list isn't fetched. There should be a message in the log files about the reason.

Best,
Sebastian

On 04/12/2017 06:55 AM, shubham.gupta wrote:
> Hey
> 
> I have around 5000 URLs in my seed Url list. If I inject the whole list, then it fails to fetch all
> documents and parse. The depth is set to 1.
> 
> But when the list is divided into a batch of 1000 URLs then it is able to fetch and parse all
> documents successfully.
> 
> In the former case 5141 URLs are injected, out of which 5127 URLs are generated and only 1300 URLs
> get fetched with status 2. Out of the rest 1342 do not have a status 2 and the rest are unfetched.
> 
> While, when the list is small, the total count of documents is 3220, out of which the documents with
> status 2 are 1298, the documents with status code other than 2 are 1922. And, the count of documents
> which have not been fetched yet is 1.
> 
> Is this a Problem of The nutch which fails to fetch huge list of URLs? Or some changes need to be
> made in the configuration files.
> 
> Please reply soon.
>