You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by mo...@richmondinformatics.com on 2006/04/20 12:17:48 UTC
java.io.IOException: Cannot create file
Hello Team,
I am using a cluster of 7 Dell SC1425's, 1 master and 6 slaves, all Xeon 3Ghz
with Gigabit ethernet and 2 x WD4000KD drives, running CentOS 4.3 and Sun
jdk1.5.0_06-b05.
Nutch/hadoop versions:
nutch-0.8 (revision 395259)
hadoop-0.2-dev (revision 395539):
My fetches (1.25M URL's) are failing during the reduce phase of an otherwise
successful run. Fetch was invoked as follows:
# nohup bin/nutch fetch /user/root/crawlA/segments/20060419162433 -threads 150
>> logs/logall.txt &
And, the errors listed below were displayed in the web ui.
First question. Updatedb won't run against the segment so what can I do to
salvage it? Is the segment salvageable? "bin/hadoop dfs -du" fills me with
hope :)
/user/root/crawlA/segments/20060419162433/content 2577482693
/user/root/crawlA/segments/20060419162433/crawl_fetch 76587779
/user/root/crawlA/segments/20060419162433/crawl_generate 91242415
/user/root/crawlA/segments/20060419162433/crawl_parse 1122074034
/user/root/crawlA/segments/20060419162433/parse_data 706562716
/user/root/crawlA/segments/20060419162433/parse_text 707691237
Second question, should I raise an issue in JIRA quoting the errors below?
Many thanks,
Monu Ogbe
*** EXHIBITS
*** Excerpt from hadoop-site.xml
<property>
<name>dfs.name.dir</name>
<value>/home/nutch/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/nutch/hadoop/dfs/data</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/nutch/hadoop/mapred/local</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/nutch/hadoop/mapred/system</value>
</property>
<property>
<name>mapred.temp.dir</name>
<value>/home/nutch/hadoop/mapred/temp</value>
</property>
*** Errors displayed in the webapp
tip_epgdt0 0.0 reduce > copy >
Timed out.java.io.IOException: Task process exit with nonzero status of 143.
at org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:273)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:145)
Timed out.java.io.IOException: Task process exit with nonzero status of 143.
at org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:273)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:145)
tip_hu8h3m 0.0
Timed out.java.io.IOException: Task process exit with nonzero status of 143.
at org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:273)
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:145)
java.rmi.RemoteException: java.io.IOException: Cannot create file
/user/root/crawlA/segments/20060419162433/parse_text/part-00005/data on client
DFSClient_task_r_poobc6
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:156)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:237)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:216)
at org.apache.hadoop.ipc.Client.call(Client.java:303)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:141)
at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:691)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:864)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:952)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at
org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:83)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:144)
at org.apache.hadoop.io.MapFile$Writer.close(MapFile.java:117)
at org.apache.nutch.parse.ParseOutputFormat$1.close(ParseOutputFormat.java:128)
at
org.apache.nutch.fetcher.FetcherOutputFormat$1.close(FetcherOutputFormat.java:96)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:290)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:743)
java.rmi.RemoteException: java.io.IOException: Cannot create file
/user/root/crawlA/segments/20060419162433/parse_text/part-00005/data on client
DFSClient_task_r_qklnfm
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:156)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:237)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:216)
at org.apache.hadoop.ipc.Client.call(Client.java:303)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:141)
at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:691)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:864)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:952)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at
org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:83)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:144)
at org.apache.hadoop.io.MapFile$Writer.close(MapFile.java:117)
at org.apache.nutch.parse.ParseOutputFormat$1.close(ParseOutputFormat.java:128)
at
org.apache.nutch.fetcher.FetcherOutputFormat$1.close(FetcherOutputFormat.java:96)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:290)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:743)
java.rmi.RemoteException: java.io.IOException: Cannot create file
/user/root/crawlA/segments/20060419162433/parse_text/part-00005/data on client
DFSClient_task_r_vu6z9t
at org.apache.hadoop.dfs.NameNode.create(NameNode.java:156)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:237)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:216)
at org.apache.hadoop.ipc.Client.call(Client.java:303)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:141)
at org.apache.hadoop.dfs.$Proxy1.create(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:691)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.endBlock(DFSClient.java:864)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:952)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at
org.apache.hadoop.fs.FSDataOutputStream$Summer.close(FSDataOutputStream.java:83)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at java.io.FilterOutputStream.close(FilterOutputStream.java:143)
at org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:144)
at org.apache.hadoop.io.MapFile$Writer.close(MapFile.java:117)
at org.apache.nutch.parse.ParseOutputFormat$1.close(ParseOutputFormat.java:128)
at
org.apache.nutch.fetcher.FetcherOutputFormat$1.close(FetcherOutputFormat.java:96)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:290)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:743)
Re: java.io.IOException: Cannot create file
Posted by Doug Cutting <cu...@apache.org>.
monu.ogbe@richmondinformatics.com wrote:
> Actually, I think that updatedb won't run because the fetched segment
> didn't
> complete correctly. Don't know whether the instructions in the 0.7 FAQ
> apply:
>
> %touch /index/segments/2005somesegment/fetcher.done
Ah. That's different. No, the 0.7 trick probably won't work.
What errors are you seeing from this? I'd expect you'd see unexpected
eof in the updatedb map task, for truncated outputs in this segment.
http://issues.apache.org/jira/browse/HADOOP-153 would fix that, once
implemented.
Doug
Re: java.io.IOException: Cannot create file
Posted by mo...@richmondinformatics.com.
Thank you Doug,
Quoting Doug Cutting <cu...@apache.org>:
> monu.ogbe@richmondinformatics.com wrote:
>> First question. Updatedb won't run against the segment so what can I do to
>> salvage it? Is the segment salvageable?
>
> Probably. I think you're hitting some current bugs in DFS &
> MapReduce. Once these are fixed, then your updatedb's should succeed!
Actually, I think that updatedb won't run because the fetched segment didn't
complete correctly. Don't know whether the instructions in the 0.7 FAQ apply:
%touch /index/segments/2005somesegment/fetcher.done
>> Second question, should I raise an issue in JIRA quoting the errors below?
>
> Yes, please.
>
Will do.
>> *** Excerpt from hadoop-site.xml
>> <property>
>> <name>mapred.system.dir</name>
>> <value>/home/nutch/hadoop/mapred/system</value>
>> </property>
>
> Unlike the other paths, mapred.system.dir is not a local path, but a
> path in the default filesystem, dfs in your case. Your setting is
> fine, I just thought I'd mention that.
>
>> Timed out.java.io.IOException: Task process exit with nonzero status of 143.
>
> These 143's are a mystery to me. We really need to figure out what
> is causing these! One suggestion I found on the net was to try
> passing '-Xrs' to java, i.e., setting mapred.child.java.opts to
> include it. Another idea is to put 'ulimit -c unlimited' in one's
> conf/hadoop-env.sh, so that these will cause core dumps. Then,
> hopefully, we can use gdb to see where the JVM crashed. I have not
> had time recently to try either of these on a cluster, the only place
> where this problem has been seen.
>
>> java.rmi.RemoteException: java.io.IOException: Cannot create file
>> /user/root/crawlA/segments/20060419162433/parse_text/part-00005/data
>> on client
>> DFSClient_task_r_poobc6
>
> This bug is triggered by the previous bug. In the first case the
> output is started, then the task jvm crashes. But DFS waits a minute
> before it will let another task create a file with the same name (to
> time out the other writer). So if the replacement task starts within
> a minute, then this error is thrown. I think Owen is working on a
> patch for this which will make DFSClient try to open the file for at
> least a minute before throwing an exception. We should have that
> committed today. This won't fix the 143's, but should allow your
> jobs to complete in spite of them.
>
> Thanks for your patience,
My patience!? I am in awe of the commitment and relentless effort of you and
the development team.
Many thanks,
Monu
Re: java.io.IOException: Cannot create file
Posted by Doug Cutting <cu...@apache.org>.
monu.ogbe@richmondinformatics.com wrote:
> First question. Updatedb won't run against the segment so what can I do to
> salvage it? Is the segment salvageable?
Probably. I think you're hitting some current bugs in DFS & MapReduce.
Once these are fixed, then your updatedb's should succeed!
> Second question, should I raise an issue in JIRA quoting the errors below?
Yes, please.
> *** Excerpt from hadoop-site.xml
> <property>
> <name>mapred.system.dir</name>
> <value>/home/nutch/hadoop/mapred/system</value>
> </property>
Unlike the other paths, mapred.system.dir is not a local path, but a
path in the default filesystem, dfs in your case. Your setting is fine,
I just thought I'd mention that.
> Timed out.java.io.IOException: Task process exit with nonzero status of 143.
These 143's are a mystery to me. We really need to figure out what is
causing these! One suggestion I found on the net was to try passing
'-Xrs' to java, i.e., setting mapred.child.java.opts to include it.
Another idea is to put 'ulimit -c unlimited' in one's
conf/hadoop-env.sh, so that these will cause core dumps. Then,
hopefully, we can use gdb to see where the JVM crashed. I have not had
time recently to try either of these on a cluster, the only place where
this problem has been seen.
> java.rmi.RemoteException: java.io.IOException: Cannot create file
> /user/root/crawlA/segments/20060419162433/parse_text/part-00005/data on client
> DFSClient_task_r_poobc6
This bug is triggered by the previous bug. In the first case the output
is started, then the task jvm crashes. But DFS waits a minute before it
will let another task create a file with the same name (to time out the
other writer). So if the replacement task starts within a minute, then
this error is thrown. I think Owen is working on a patch for this which
will make DFSClient try to open the file for at least a minute before
throwing an exception. We should have that committed today. This won't
fix the 143's, but should allow your jobs to complete in spite of them.
Thanks for your patience,
Doug