You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2006/01/24 23:20:10 UTC
[jira] Closed: (NUTCH-136) mapreduce segment generator generates
50 % less than excepted urls
[ http://issues.apache.org/jira/browse/NUTCH-136?page=all ]
Andrzej Bialecki closed NUTCH-136:
-----------------------------------
Resolution: Duplicate
Thank you for investigating this. I'm closing this issue, further discussion should follow to NUTCH-186.
> mapreduce segment generator generates 50 % less than excepted urls
> --------------------------------------------------------------------
>
> Key: NUTCH-136
> URL: http://issues.apache.org/jira/browse/NUTCH-136
> Project: Nutch
> Type: Bug
> Versions: 0.8-dev
> Reporter: Stefan Groschupf
> Priority: Critical
>
> We notice that segments generated with the map reduce segment generator contains only 50 % of the expected urls. We had a crawldb with 40 000 urls and the generate commands only created a 20 000 pages segment. This also happened with the topN parameter, we everytime got around 50 % of the expected urls.
> I tested the PartitionUrlByHost and it looks like it does its work. However we fixed the problem by changing two things:
> First we set the partition to a normal hashPartitioner.
> Second we changed Generator.java line 48:
> limit = job.getLong("crawl.topN",Long.MAX_VALUE)/job.getNumReduceTasks();
> to:
> limit = job.getLong("crawl.topN",Long.MAX_VALUE);
> Now it works as expected.
> Has anyone a idea what the real source of this problem can be?
> In general this is bug has the effect that all map reduce users fetch only 50 % of it's urls per iteration.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira