You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Rafael Pappert <ra...@pappert.biz> on 2012/03/16 14:46:19 UTC
Blacklisted Tasktracker / AlreadyBeingCreatedException
Hello,
I'm running nutch 1.4 on an 3 Node Hadoop Cluster and from time to
time i got an "alert" that 1 TaskTracker have been blacklisted.
And the log of the reducer contains 3-6 Exceptions like this:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/test/crawl/segments/20120316065507/parse_text/part-00001/data for DFSClient_attempt_201203151054_0028_r_000001_1 on client xx.x.xx.xx.10, because this file is already being created by DFSClient_attempt_201203151054_0028_r_000001_0 on xx.xx.xx.9
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1404)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1244)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1186)
at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
at org.apache.hadoop.ipc.Client.call(Client.java:1066)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at $Proxy2.create(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy2.create(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3245)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:182)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile.java:1132)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:157)
at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:134)
at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:92)
at org.apache.nutch.parse.ParseOutputFormat.getRecordWriter(ParseOutputFormat.java:110)
at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:448)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:490)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
I have no special Plugins, it's a "default" system. Any ideas?
Thanks in advance,
Rafael.
Re: Blacklisted Tasktracker / AlreadyBeingCreatedException
Posted by Mathijs Homminga <ma...@kalooga.com>.
Hi Rafael,
This sounds like a Hadoop DFS issue. Perhaps it's better to post your question to:
hdfs-user@hadoop.apache.org
Mathijs
On 16 mrt. 2012, at 14:46, Rafael Pappert wrote:
> Hello,
>
> I'm running nutch 1.4 on an 3 Node Hadoop Cluster and from time to
> time i got an "alert" that 1 TaskTracker have been blacklisted.
>
> And the log of the reducer contains 3-6 Exceptions like this:
>
> org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to create file /user/test/crawl/segments/20120316065507/parse_text/part-00001/data for DFSClient_attempt_201203151054_0028_r_000001_1 on client xx.x.xx.xx.10, because this file is already being created by DFSClient_attempt_201203151054_0028_r_000001_0 on xx.xx.xx.9
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:1404)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1244)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1186)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1066)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy2.create(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy2.create(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3245)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
> at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:182)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile.java:1132)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476)
> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:157)
> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:134)
> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:92)
> at org.apache.nutch.parse.ParseOutputFormat.getRecordWriter(ParseOutputFormat.java:110)
> at org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceTask.java:448)
> at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:490)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> I have no special Plugins, it's a "default" system. Any ideas?
>
> Thanks in advance,
> Rafael.
Re: Blacklisted Tasktracker / AlreadyBeingCreatedException
Posted by Rafael Pappert <rp...@fwpsystems.com>.
Hi Lewis,
mission accomplished :)
https://issues.apache.org/jira/browse/NUTCH-1315
On 19/Mar/ 2012, at 13:53 , Lewis John Mcgibbney wrote:
> Hi Rafael,
>
> This is an interesting one and in all honesty off the top of my head I
> don't know.
> I would urge you to open a ticket (after taking some time to see if
> anything already exists).
>
> If you could do this it would be great and would give us somewhere to log
> the progress with the issue.
>
> Thank you
>
> On Mon, Mar 19, 2012 at 9:26 AM, Rafael Pappert <rp...@fwpsystems.com> wrote:
>
>> Hello,
>>
>> I wrote to the hdfs-user mailing list and got the following answer:
>>
>> "Looks like you have reduce speculation turned on, but the
>> ParseOutputFormat you're using doesn't properly name its output files
>> distinctly based on the task attempt ID. As a workaround you can
>> probably turn off speculative execution for reduces, but you should
>> also probably file a Nutch bug."
>>
>> Is reduce speculation turned on by nutch? My configuration is
>> "default" an I didn't turn it on manually.
>>
>> cheers,
>> rafael.
>>
>>
>>
>> On 16/Mar/ 2012, at 15:13 , Markus Jelsma wrote:
>>
>>> Very nasty! Please consult the Hadoop list and tell which version you are
>>> running and if you have any interesting non-default configuration
>> options set.
>>>
>>> On Friday 16 March 2012 14:46:19 Rafael Pappert wrote:
>>>> Hello,
>>>>
>>>> I'm running nutch 1.4 on an 3 Node Hadoop Cluster and from time to
>>>> time i got an "alert" that 1 TaskTracker have been blacklisted.
>>>>
>>>> And the log of the reducer contains 3-6 Exceptions like this:
>>>>
>>>> org.apache.hadoop.ipc.RemoteException:
>>>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>>>> create file
>>>> /user/test/crawl/segments/20120316065507/parse_text/part-00001/data for
>>>> DFSClient_attempt_201203151054_0028_r_000001_1 on client xx.x.xx.xx.10,
>>>> because this file is already being created by
>>>> DFSClient_attempt_201203151054_0028_r_000001_0 on xx.xx.xx.9 at
>>>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(F
>>>> SNamesystem.java:1404) at
>>>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNa
>>>> mesystem.java:1244) at
>>>>
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem
>>>> .java:1186) at
>>>>
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)
>>>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>>>> at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>>> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>>> java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1066)
>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>>>> at $Proxy2.create(Unknown Source)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>>>> 39) at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>>>> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at
>>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoca
>>>> tionHandler.java:82) at
>>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHa
>>>> ndler.java:59) at $Proxy2.create(Unknown Source)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:324
>>>> 5) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
>>>> at
>>>>
>> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.
>>>> java:182) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>>>> at
>>>>
>> org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile
>>>> .java:1132) at
>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397) at
>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354) at
>>>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476) at
>>>> org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:157)
>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:134)
>>>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:92)
>>>> at
>>>>
>> org.apache.nutch.parse.ParseOutputFormat.getRecordWriter(ParseOutputFormat
>>>> .java:110) at
>>>>
>> org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceT
>>>> ask.java:448) at
>>>> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:490)
>> at
>>>> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>>>> java:1093) at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>>
>>>> I have no special Plugins, it's a "default" system. Any ideas?
>>>>
>>>> Thanks in advance,
>>>> Rafael.
>>>
>>> --
>>> Markus Jelsma - CTO - Openindex
>>
>>
>
>
> --
> *Lewis*
Re: Blacklisted Tasktracker / AlreadyBeingCreatedException
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Rafael,
This is an interesting one and in all honesty off the top of my head I
don't know.
I would urge you to open a ticket (after taking some time to see if
anything already exists).
If you could do this it would be great and would give us somewhere to log
the progress with the issue.
Thank you
On Mon, Mar 19, 2012 at 9:26 AM, Rafael Pappert <rp...@fwpsystems.com> wrote:
> Hello,
>
> I wrote to the hdfs-user mailing list and got the following answer:
>
> "Looks like you have reduce speculation turned on, but the
> ParseOutputFormat you're using doesn't properly name its output files
> distinctly based on the task attempt ID. As a workaround you can
> probably turn off speculative execution for reduces, but you should
> also probably file a Nutch bug."
>
> Is reduce speculation turned on by nutch? My configuration is
> "default" an I didn't turn it on manually.
>
> cheers,
> rafael.
>
>
>
> On 16/Mar/ 2012, at 15:13 , Markus Jelsma wrote:
>
> > Very nasty! Please consult the Hadoop list and tell which version you are
> > running and if you have any interesting non-default configuration
> options set.
> >
> > On Friday 16 March 2012 14:46:19 Rafael Pappert wrote:
> >> Hello,
> >>
> >> I'm running nutch 1.4 on an 3 Node Hadoop Cluster and from time to
> >> time i got an "alert" that 1 TaskTracker have been blacklisted.
> >>
> >> And the log of the reducer contains 3-6 Exceptions like this:
> >>
> >> org.apache.hadoop.ipc.RemoteException:
> >> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
> >> create file
> >> /user/test/crawl/segments/20120316065507/parse_text/part-00001/data for
> >> DFSClient_attempt_201203151054_0028_r_000001_1 on client xx.x.xx.xx.10,
> >> because this file is already being created by
> >> DFSClient_attempt_201203151054_0028_r_000001_0 on xx.xx.xx.9 at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(F
> >> SNamesystem.java:1404) at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNa
> >> mesystem.java:1244) at
> >>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem
> >> .java:1186) at
> >>
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)
> >> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
> >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:396)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> >> java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
> >>
> >> at org.apache.hadoop.ipc.Client.call(Client.java:1066)
> >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> >> at $Proxy2.create(Unknown Source)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >> 39) at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> >> at
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoca
> >> tionHandler.java:82) at
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHa
> >> ndler.java:59) at $Proxy2.create(Unknown Source)
> >> at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:324
> >> 5) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
> >> at
> >>
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.
> >> java:182) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> >> at
> >>
> org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile
> >> .java:1132) at
> >> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397) at
> >> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354) at
> >> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476) at
> >> org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:157)
> >> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:134)
> >> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:92)
> >> at
> >>
> org.apache.nutch.parse.ParseOutputFormat.getRecordWriter(ParseOutputFormat
> >> .java:110) at
> >>
> org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceT
> >> ask.java:448) at
> >> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:490)
> at
> >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:396)
> >> at
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> >> java:1093) at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >>
> >> I have no special Plugins, it's a "default" system. Any ideas?
> >>
> >> Thanks in advance,
> >> Rafael.
> >
> > --
> > Markus Jelsma - CTO - Openindex
>
>
--
*Lewis*
Re: Blacklisted Tasktracker / AlreadyBeingCreatedException
Posted by Rafael Pappert <rp...@fwpsystems.com>.
Hello,
I wrote to the hdfs-user mailing list and got the following answer:
"Looks like you have reduce speculation turned on, but the
ParseOutputFormat you're using doesn't properly name its output files
distinctly based on the task attempt ID. As a workaround you can
probably turn off speculative execution for reduces, but you should
also probably file a Nutch bug."
Is reduce speculation turned on by nutch? My configuration is
"default" an I didn't turn it on manually.
cheers,
rafael.
On 16/Mar/ 2012, at 15:13 , Markus Jelsma wrote:
> Very nasty! Please consult the Hadoop list and tell which version you are
> running and if you have any interesting non-default configuration options set.
>
> On Friday 16 March 2012 14:46:19 Rafael Pappert wrote:
>> Hello,
>>
>> I'm running nutch 1.4 on an 3 Node Hadoop Cluster and from time to
>> time i got an "alert" that 1 TaskTracker have been blacklisted.
>>
>> And the log of the reducer contains 3-6 Exceptions like this:
>>
>> org.apache.hadoop.ipc.RemoteException:
>> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
>> create file
>> /user/test/crawl/segments/20120316065507/parse_text/part-00001/data for
>> DFSClient_attempt_201203151054_0028_r_000001_1 on client xx.x.xx.xx.10,
>> because this file is already being created by
>> DFSClient_attempt_201203151054_0028_r_000001_0 on xx.xx.xx.9 at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(F
>> SNamesystem.java:1404) at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNa
>> mesystem.java:1244) at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem
>> .java:1186) at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)
>> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>> java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1066)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
>> at $Proxy2.create(Unknown Source)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>> 39) at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
>> at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoca
>> tionHandler.java:82) at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHa
>> ndler.java:59) at $Proxy2.create(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:324
>> 5) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.
>> java:182) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>> at
>> org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile
>> .java:1132) at
>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397) at
>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354) at
>> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476) at
>> org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:157)
>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:134)
>> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:92)
>> at
>> org.apache.nutch.parse.ParseOutputFormat.getRecordWriter(ParseOutputFormat
>> .java:110) at
>> org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceT
>> ask.java:448) at
>> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:490) at
>> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>> java:1093) at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>
>> I have no special Plugins, it's a "default" system. Any ideas?
>>
>> Thanks in advance,
>> Rafael.
>
> --
> Markus Jelsma - CTO - Openindex
Re: Blacklisted Tasktracker / AlreadyBeingCreatedException
Posted by Markus Jelsma <ma...@openindex.io>.
Very nasty! Please consult the Hadoop list and tell which version you are
running and if you have any interesting non-default configuration options set.
On Friday 16 March 2012 14:46:19 Rafael Pappert wrote:
> Hello,
>
> I'm running nutch 1.4 on an 3 Node Hadoop Cluster and from time to
> time i got an "alert" that 1 TaskTracker have been blacklisted.
>
> And the log of the reducer contains 3-6 Exceptions like this:
>
> org.apache.hadoop.ipc.RemoteException:
> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to
> create file
> /user/test/crawl/segments/20120316065507/parse_text/part-00001/data for
> DFSClient_attempt_201203151054_0028_r_000001_1 on client xx.x.xx.xx.10,
> because this file is already being created by
> DFSClient_attempt_201203151054_0028_r_000001_0 on xx.xx.xx.9 at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(F
> SNamesystem.java:1404) at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNa
> mesystem.java:1244) at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem
> .java:1186) at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:628)
> at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> java:1093) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1066)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
> at $Proxy2.create(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 39) at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> pl.java:25) at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoca
> tionHandler.java:82) at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHa
> ndler.java:59) at $Proxy2.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:324
> 5) at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:713)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.
> java:182) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> at
> org.apache.hadoop.io.SequenceFile$RecordCompressWriter.<init>(SequenceFile
> .java:1132) at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:397) at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:354) at
> org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:476) at
> org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:157)
> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:134)
> at org.apache.hadoop.io.MapFile$Writer.<init>(MapFile.java:92)
> at
> org.apache.nutch.parse.ParseOutputFormat.getRecordWriter(ParseOutputFormat
> .java:110) at
> org.apache.hadoop.mapred.ReduceTask$OldTrackingRecordWriter.<init>(ReduceT
> ask.java:448) at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:490) at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
> java:1093) at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> I have no special Plugins, it's a "default" system. Any ideas?
>
> Thanks in advance,
> Rafael.
--
Markus Jelsma - CTO - Openindex