You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by MING-Yuan JIANG <cn...@gmail.com> on 2009/12/17 09:00:28 UTC

Convert Arc file to segement with ArcSegmentCreator,run very slow

Hi everyone

When I convert arc file to segment with the tool ArcSegmentCreator.I found
it run very slow.

My hadoop-site.xml modified someplace as below:

dfs.block.size  50MB
mapred.reduce.tasks 160
mapred.tasktracker.map.tasks.maximum 8
mapred.tasktracker.reduce.tasks.maximum 8

And I have 2 tasktracker,per node has 2 cores.
All the data in hdfs are 100 arc files,each file 100MB.

How to set those parameters above to make job run faster?

thanks!