You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Justin Yao <ju...@snooth.com> on 2009/03/05 22:57:13 UTC
Error on merging segments
Hi,
I encountered an error when merging segment using nightly build #736.
I have 3 nodes and all servers have CentOS 5.2 installed.
Every time when I tried to merge segment using command
"nutch mergesegs crawl/MERGEDsegments -dir crawl/segments",
it will failed with error message
"Task attempt_200903031109_0007_r_000002_0 failed to report status for
603 seconds. Killing!"
and later
"org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
create file xxxxxxxx/crawl_generate/part-00002 for
DFSClient_attempt_200903031109_0007_r_000002_2 on client 10.6.180.2
because current leaseholder is trying to recreate file"
If I raised "mapred.task.timeout" to "3600000" (1 hour), it will still fail.
I've tried build #743, #736 and #723. None will succeed.
Does someone encounter this problem before? Any suggestion would be
appreciated.
Thanks,
--
Justin Yao
Re: Error on merging segments
Posted by Justin Yao <ju...@snooth.com>.
I think I find the solution.
I raised maximum heap size setting in mapred.child.java.opts, lowered
mapred.tasktracker.reduce.tasks.maximum and started the segment merging
task again, it succeeded this time.
Thanks,
Justin
Justin Yao wrote:
> Sorry, I didn't notice the "xxxxxxxx" masks.
>
> where "xxxxxxxx" in xxxxxxxx/crawl_generate/part-00002
> is
> /user/justin/crawl/MERGEDsegments
>
> Thanks,
> Justin
>
> Justin Yao wrote:
>> Hi,
>>
>> I encountered an error when merging segment using nightly build #736.
>> I have 3 nodes and all servers have CentOS 5.2 installed.
>> Every time when I tried to merge segment using command
>> "nutch mergesegs crawl/MERGEDsegments -dir crawl/segments",
>> it will failed with error message
>> "Task attempt_200903031109_0007_r_000002_0 failed to report status for
>> 603 seconds. Killing!"
>> and later
>> "org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>> create file xxxxxxxx/crawl_generate/part-00002 for
>> DFSClient_attempt_200903031109_0007_r_000002_2 on client 10.6.180.2
>> because current leaseholder is trying to recreate file"
>>
>> If I raised "mapred.task.timeout" to "3600000" (1 hour), it will still
>> fail.
>>
>> I've tried build #743, #736 and #723. None will succeed.
>> Does someone encounter this problem before? Any suggestion would be
>> appreciated.
>>
>> Thanks,
Re: Error on merging segments
Posted by Justin Yao <ju...@snooth.com>.
Sorry, I didn't notice the "xxxxxxxx" masks.
where "xxxxxxxx" in xxxxxxxx/crawl_generate/part-00002
is
/user/justin/crawl/MERGEDsegments
Thanks,
Justin
Justin Yao wrote:
> Hi,
>
> I encountered an error when merging segment using nightly build #736.
> I have 3 nodes and all servers have CentOS 5.2 installed.
> Every time when I tried to merge segment using command
> "nutch mergesegs crawl/MERGEDsegments -dir crawl/segments",
> it will failed with error message
> "Task attempt_200903031109_0007_r_000002_0 failed to report status for
> 603 seconds. Killing!"
> and later
> "org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
> create file xxxxxxxx/crawl_generate/part-00002 for
> DFSClient_attempt_200903031109_0007_r_000002_2 on client 10.6.180.2
> because current leaseholder is trying to recreate file"
>
> If I raised "mapred.task.timeout" to "3600000" (1 hour), it will still
> fail.
>
> I've tried build #743, #736 and #723. None will succeed.
> Does someone encounter this problem before? Any suggestion would be
> appreciated.
>
> Thanks,