You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Justin Yao <ju...@snooth.com> on 2009/03/05 22:57:13 UTC

Error on merging segments

Hi,

I encountered an error when merging segment using nightly build #736.
I have 3 nodes and all servers have CentOS 5.2 installed.
Every time when I tried to merge segment using command
"nutch mergesegs crawl/MERGEDsegments -dir crawl/segments",
it will failed with error message
"Task attempt_200903031109_0007_r_000002_0 failed to report status for
603 seconds. Killing!"
and later
"org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
create file xxxxxxxx/crawl_generate/part-00002 for
DFSClient_attempt_200903031109_0007_r_000002_2 on client 10.6.180.2
because current leaseholder is trying to recreate file"

If I raised "mapred.task.timeout" to "3600000" (1 hour), it will still fail.

I've tried build #743, #736 and #723. None will succeed.
Does someone encounter this problem before? Any suggestion would be 
appreciated.

Thanks,
-- 
Justin Yao




Re: Error on merging segments

Posted by Justin Yao <ju...@snooth.com>.
I think I find the solution.
I raised maximum heap size setting in mapred.child.java.opts, lowered 
mapred.tasktracker.reduce.tasks.maximum and started the segment merging 
task again, it succeeded this time.

Thanks,
Justin

Justin Yao wrote:
> Sorry, I didn't notice the "xxxxxxxx" masks.
> 
> where "xxxxxxxx" in xxxxxxxx/crawl_generate/part-00002
> is
> /user/justin/crawl/MERGEDsegments
> 
> Thanks,
> Justin
> 
> Justin Yao wrote:
>> Hi,
>>
>> I encountered an error when merging segment using nightly build #736.
>> I have 3 nodes and all servers have CentOS 5.2 installed.
>> Every time when I tried to merge segment using command
>> "nutch mergesegs crawl/MERGEDsegments -dir crawl/segments",
>> it will failed with error message
>> "Task attempt_200903031109_0007_r_000002_0 failed to report status for
>> 603 seconds. Killing!"
>> and later
>> "org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>> create file xxxxxxxx/crawl_generate/part-00002 for
>> DFSClient_attempt_200903031109_0007_r_000002_2 on client 10.6.180.2
>> because current leaseholder is trying to recreate file"
>>
>> If I raised "mapred.task.timeout" to "3600000" (1 hour), it will still 
>> fail.
>>
>> I've tried build #743, #736 and #723. None will succeed.
>> Does someone encounter this problem before? Any suggestion would be 
>> appreciated.
>>
>> Thanks,

Re: Error on merging segments

Posted by Justin Yao <ju...@snooth.com>.
Sorry, I didn't notice the "xxxxxxxx" masks.

where "xxxxxxxx" in xxxxxxxx/crawl_generate/part-00002
is
/user/justin/crawl/MERGEDsegments

Thanks,
Justin

Justin Yao wrote:
> Hi,
> 
> I encountered an error when merging segment using nightly build #736.
> I have 3 nodes and all servers have CentOS 5.2 installed.
> Every time when I tried to merge segment using command
> "nutch mergesegs crawl/MERGEDsegments -dir crawl/segments",
> it will failed with error message
> "Task attempt_200903031109_0007_r_000002_0 failed to report status for
> 603 seconds. Killing!"
> and later
> "org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
> create file xxxxxxxx/crawl_generate/part-00002 for
> DFSClient_attempt_200903031109_0007_r_000002_2 on client 10.6.180.2
> because current leaseholder is trying to recreate file"
> 
> If I raised "mapred.task.timeout" to "3600000" (1 hour), it will still 
> fail.
> 
> I've tried build #743, #736 and #723. None will succeed.
> Does someone encounter this problem before? Any suggestion would be 
> appreciated.
> 
> Thanks,