You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Eric Osgood <er...@lakemeadonline.com> on 2009/10/20 23:00:47 UTC
ERROR: current leaseholder is trying to recreate file.
This is the error I keep getting whenever I try to fetch more than
400K files at a time using a 4 node hadoop cluster running nutch 1.0.
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed
to create file /user/hadoop/crawl/segments/20091013161641/crawl_fetch/
part-00015/index for DFSClient_attempt_200910131302_0011_r_000015_2 on
client 192.168.1.201 because current leaseholder is trying to recreate
file.
Can anybody shed some light on this issue? I was under the impression
that 400K was small potatoes for a nutch hadoop combo?
Thanks,
Eric Osgood
---------------------------------------------
Cal Poly - Computer Engineering, Moon Valley Software
---------------------------------------------
eosgood@calpoly.edu, eric@lakemeadonline.com
---------------------------------------------
www.calpoly.edu/~eosgood, www.lakemeadonline.com
Re: ERROR: current leaseholder is trying to recreate file.
Posted by Eric Osgood <er...@lakemeadonline.com>.
Andrzej,
I just downloaded the most recent trunk from svn as per your
recommendations for fixing the generate bug. As soon I have it all
rebuilt with my configs I will let you know how a crawl of ~1.6mln
pages goes. Hopefully no errors!
Thanks,
Eric
On Oct 20, 2009, at 2:13 PM, Andrzej Bialecki wrote:
> Eric Osgood wrote:
>> This is the error I keep getting whenever I try to fetch more than
>> 400K files at a time using a 4 node hadoop cluster running nutch 1.0.
>> org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
>> failed to create file /user/hadoop/crawl/segments/20091013161641/
>> crawl_fetch/part-00015/index for
>> DFSClient_attempt_200910131302_0011_r_000015_2 on client
>> 192.168.1.201 because current leaseholder is trying to recreate file.
>
> Please see this issue:
>
> https://issues.apache.org/jira/browse/NUTCH-692
>
> Apply the patch that is attached there, rebuild Nutch, and tell me
> if this fixes your problem.
>
> (the patch will be applied to trunk anyway, since others confirmed
> that it fixes this issue).
>
>> Can anybody shed some light on this issue? I was under the
>> impression that 400K was small potatoes for a nutch hadoop combo?
>
> It is. This problem is rare - I think I crawled cumulatively ~500mln
> pages in various configs and it didn't occur to me personally. It
> requires a few things to go wrong (see the issue comments).
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
Eric Osgood
---------------------------------------------
Cal Poly - Computer Engineering, Moon Valley Software
---------------------------------------------
eosgood@calpoly.edu, eric@lakemeadonline.com
---------------------------------------------
www.calpoly.edu/~eosgood, www.lakemeadonline.com
Re: ERROR: current leaseholder is trying to recreate file.
Posted by Eric Osgood <er...@lakemeadonline.com>.
Andrzej,
I updated nutch to the trunk last night and I split up a crawl of 1.6M
into 4 chunks of 400K using the updated generator. However, the first
crawl of 400K crashed last night with some new errors I have never
seen before:
org.apache.hadoop.fs.ChecksumException: Checksum Error
java.io.IOException: Could not obtain block:
blk_-8206810763586975866_5190 file=/user/hadoop/crawl/segments/
20091020170107/crawl_generate/part-00009
Do you know why I would be getting these errors? I had a lost tracker
error also - could these problems be related?
Thanks,
Eric
On Oct 20, 2009, at 2:13 PM, Andrzej Bialecki wrote:
> Eric Osgood wrote:
>> This is the error I keep getting whenever I try to fetch more than
>> 400K files at a time using a 4 node hadoop cluster running nutch 1.0.
>> org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException:
>> failed to create file /user/hadoop/crawl/segments/20091013161641/
>> crawl_fetch/part-00015/index for
>> DFSClient_attempt_200910131302_0011_r_000015_2 on client
>> 192.168.1.201 because current leaseholder is trying to recreate file.
>
> Please see this issue:
>
> https://issues.apache.org/jira/browse/NUTCH-692
>
> Apply the patch that is attached there, rebuild Nutch, and tell me
> if this fixes your problem.
>
> (the patch will be applied to trunk anyway, since others confirmed
> that it fixes this issue).
>
>> Can anybody shed some light on this issue? I was under the
>> impression that 400K was small potatoes for a nutch hadoop combo?
>
> It is. This problem is rare - I think I crawled cumulatively ~500mln
> pages in various configs and it didn't occur to me personally. It
> requires a few things to go wrong (see the issue comments).
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
Eric Osgood
---------------------------------------------
Cal Poly - Computer Engineering, Moon Valley Software
---------------------------------------------
eosgood@calpoly.edu, eric@lakemeadonline.com
---------------------------------------------
www.calpoly.edu/~eosgood, www.lakemeadonline.com
Re: ERROR: current leaseholder is trying to recreate file.
Posted by Andrzej Bialecki <ab...@getopt.org>.
Eric Osgood wrote:
> This is the error I keep getting whenever I try to fetch more than 400K
> files at a time using a 4 node hadoop cluster running nutch 1.0.
>
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
> create file
> /user/hadoop/crawl/segments/20091013161641/crawl_fetch/part-00015/index
> for DFSClient_attempt_200910131302_0011_r_000015_2 on client
> 192.168.1.201 because current leaseholder is trying to recreate file.
Please see this issue:
https://issues.apache.org/jira/browse/NUTCH-692
Apply the patch that is attached there, rebuild Nutch, and tell me if
this fixes your problem.
(the patch will be applied to trunk anyway, since others confirmed that
it fixes this issue).
>
> Can anybody shed some light on this issue? I was under the impression
> that 400K was small potatoes for a nutch hadoop combo?
It is. This problem is rare - I think I crawled cumulatively ~500mln
pages in various configs and it didn't occur to me personally. It
requires a few things to go wrong (see the issue comments).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com