You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by AnilKumar B <ak...@gmail.com> on 2013/12/30 11:52:15 UTC
Job fails while re attempting the task in multiple outputs case
Hi,
I am using multiple outputs in our job. So whenever any reduce task fails,
all it's next task attempts are failing with file exist exception.
The output file name should also append the task attempt right? But it's
only appending the task id. Is this the bug or Some thing wrong from my
side?
Where should look in src code? I went through code at
FileOutputFormat$getTaskOutputPath(), but there it's only considering task
id.
Exception Trace:
13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
attempt_201312162255_60465_r_000008_0, Status : FAILED
13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
attempt_201312162255_60465_r_000008_1, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
create /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client
10.103.10.31 either because the filename is invalid or the file exists
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
at org.apache.hadoop.ipc.Client.call(Client.java:1118)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at $Proxy7.create(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at $Proxy7.create(Unknown Source)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
at
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
at
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
at
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
at
org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
at
org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
at
com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
at
com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
attempt_201312162255_60465_r_000008_2, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
create /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client
10.103.7.33 either because the filename is invalid or the file exists
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
Thanks & Regards,
B Anil Kumar.
Re: Job fails while re attempting the task in multiple outputs case
Posted by Jiayu Ji <ji...@gmail.com>.
I think if the task fails, the output related to that task will be clean up
before the second attempt. I am guessing you have this exception is because
you have two reducers tried to write to the same file. One thing you need
to be aware of is that all data that is supposed to be in the same file
should go to the same reducer. Let's say if you have data1 on reducer1 and
data2 on reducer2 and they are all going to be stored on fileAll, then you
will end up having that exception.
On Mon, Dec 30, 2013 at 11:22 AM, AnilKumar B <ak...@gmail.com> wrote:
> Thanks Harsh.
>
> @Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
> I am using Apache Hadoop's multipleOutputs.
>
> But as you see in stack trace, it's not appending the attempt id to file
> name, it's only consists of task id.
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Are you using the MultipleOutputs class shipped with Apache Hadoop or
>> one of your own?
>>
>> If its the latter, please take a look at gotchas to take care of
>> described at
>> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>>
>> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am using multiple outputs in our job. So whenever any reduce task
>> fails,
>> > all it's next task attempts are failing with file exist exception.
>> >
>> >
>> > The output file name should also append the task attempt right? But it's
>> > only appending the task id. Is this the bug or Some thing wrong from my
>> > side?
>> >
>> > Where should look in src code? I went through code at
>> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
>> task
>> > id.
>> >
>> >
>> > Exception Trace:
>> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_0, Status : FAILED
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
>> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
>> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_1, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
>> > either because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
>> > at
>> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> > at $Proxy7.create(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>> > at $Proxy7.create(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
>> > at
>> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
>> > at
>> >
>> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
>> > at
>> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> >
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
>> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
>> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
>> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_2, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
>> either
>> > because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> >
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Jiayu (James) Ji,
Cell: (312)823-7393
Re: Job fails while re attempting the task in multiple outputs case
Posted by Jiayu Ji <ji...@gmail.com>.
I think if the task fails, the output related to that task will be clean up
before the second attempt. I am guessing you have this exception is because
you have two reducers tried to write to the same file. One thing you need
to be aware of is that all data that is supposed to be in the same file
should go to the same reducer. Let's say if you have data1 on reducer1 and
data2 on reducer2 and they are all going to be stored on fileAll, then you
will end up having that exception.
On Mon, Dec 30, 2013 at 11:22 AM, AnilKumar B <ak...@gmail.com> wrote:
> Thanks Harsh.
>
> @Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
> I am using Apache Hadoop's multipleOutputs.
>
> But as you see in stack trace, it's not appending the attempt id to file
> name, it's only consists of task id.
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Are you using the MultipleOutputs class shipped with Apache Hadoop or
>> one of your own?
>>
>> If its the latter, please take a look at gotchas to take care of
>> described at
>> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>>
>> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am using multiple outputs in our job. So whenever any reduce task
>> fails,
>> > all it's next task attempts are failing with file exist exception.
>> >
>> >
>> > The output file name should also append the task attempt right? But it's
>> > only appending the task id. Is this the bug or Some thing wrong from my
>> > side?
>> >
>> > Where should look in src code? I went through code at
>> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
>> task
>> > id.
>> >
>> >
>> > Exception Trace:
>> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_0, Status : FAILED
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
>> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
>> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_1, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
>> > either because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
>> > at
>> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> > at $Proxy7.create(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>> > at $Proxy7.create(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
>> > at
>> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
>> > at
>> >
>> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
>> > at
>> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> >
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
>> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
>> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
>> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_2, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
>> either
>> > because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> >
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Jiayu (James) Ji,
Cell: (312)823-7393
Re: Job fails while re attempting the task in multiple outputs case
Posted by Jiayu Ji <ji...@gmail.com>.
I think if the task fails, the output related to that task will be clean up
before the second attempt. I am guessing you have this exception is because
you have two reducers tried to write to the same file. One thing you need
to be aware of is that all data that is supposed to be in the same file
should go to the same reducer. Let's say if you have data1 on reducer1 and
data2 on reducer2 and they are all going to be stored on fileAll, then you
will end up having that exception.
On Mon, Dec 30, 2013 at 11:22 AM, AnilKumar B <ak...@gmail.com> wrote:
> Thanks Harsh.
>
> @Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
> I am using Apache Hadoop's multipleOutputs.
>
> But as you see in stack trace, it's not appending the attempt id to file
> name, it's only consists of task id.
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Are you using the MultipleOutputs class shipped with Apache Hadoop or
>> one of your own?
>>
>> If its the latter, please take a look at gotchas to take care of
>> described at
>> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>>
>> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am using multiple outputs in our job. So whenever any reduce task
>> fails,
>> > all it's next task attempts are failing with file exist exception.
>> >
>> >
>> > The output file name should also append the task attempt right? But it's
>> > only appending the task id. Is this the bug or Some thing wrong from my
>> > side?
>> >
>> > Where should look in src code? I went through code at
>> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
>> task
>> > id.
>> >
>> >
>> > Exception Trace:
>> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_0, Status : FAILED
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
>> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
>> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_1, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
>> > either because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
>> > at
>> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> > at $Proxy7.create(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>> > at $Proxy7.create(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
>> > at
>> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
>> > at
>> >
>> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
>> > at
>> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> >
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
>> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
>> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
>> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_2, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
>> either
>> > because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> >
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Jiayu (James) Ji,
Cell: (312)823-7393
Re: Job fails while re attempting the task in multiple outputs case
Posted by Jiayu Ji <ji...@gmail.com>.
I think if the task fails, the output related to that task will be clean up
before the second attempt. I am guessing you have this exception is because
you have two reducers tried to write to the same file. One thing you need
to be aware of is that all data that is supposed to be in the same file
should go to the same reducer. Let's say if you have data1 on reducer1 and
data2 on reducer2 and they are all going to be stored on fileAll, then you
will end up having that exception.
On Mon, Dec 30, 2013 at 11:22 AM, AnilKumar B <ak...@gmail.com> wrote:
> Thanks Harsh.
>
> @Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
> I am using Apache Hadoop's multipleOutputs.
>
> But as you see in stack trace, it's not appending the attempt id to file
> name, it's only consists of task id.
>
>
>
> Thanks & Regards,
> B Anil Kumar.
>
>
> On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Are you using the MultipleOutputs class shipped with Apache Hadoop or
>> one of your own?
>>
>> If its the latter, please take a look at gotchas to take care of
>> described at
>> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>>
>> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am using multiple outputs in our job. So whenever any reduce task
>> fails,
>> > all it's next task attempts are failing with file exist exception.
>> >
>> >
>> > The output file name should also append the task attempt right? But it's
>> > only appending the task id. Is this the bug or Some thing wrong from my
>> > side?
>> >
>> > Where should look in src code? I went through code at
>> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
>> task
>> > id.
>> >
>> >
>> > Exception Trace:
>> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_0, Status : FAILED
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
>> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
>> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
>> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_1, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
>> > either because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
>> > at
>> org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> > at $Proxy7.create(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>> > at $Proxy7.create(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
>> > at
>> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
>> > at
>> >
>> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
>> > at
>> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
>> > at
>> >
>> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
>> > at
>> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
>> > at
>> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>> > at
>> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> > at java.security.AccessController.doPrivileged(Native
>> > Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> >
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
>> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
>> >
>> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
>> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
>> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
>> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
>> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
>> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
>> > attempt_201312162255_60465_r_000008_2, Status : FAILED
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
>> create
>> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
>> either
>> > because the filename is invalid or the file exists
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
>> > at
>> >
>> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
>> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
>> > Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>> > at
>> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>> >
>> >
>> > Thanks & Regards,
>> > B Anil Kumar.
>>
>>
>>
>> --
>> Harsh J
>>
>
>
--
Jiayu (James) Ji,
Cell: (312)823-7393
Re: Job fails while re attempting the task in multiple outputs case
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Harsh.
@Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
I am using Apache Hadoop's multipleOutputs.
But as you see in stack trace, it's not appending the attempt id to file
name, it's only consists of task id.
Thanks & Regards,
B Anil Kumar.
On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
> Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
>
> If its the latter, please take a look at gotchas to take care of
> described at
> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>
> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
> wrote:
> > Hi,
> >
> > I am using multiple outputs in our job. So whenever any reduce task
> fails,
> > all it's next task attempts are failing with file exist exception.
> >
> >
> > The output file name should also append the task attempt right? But it's
> > only appending the task id. Is this the bug or Some thing wrong from my
> > side?
> >
> > Where should look in src code? I went through code at
> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
> task
> > id.
> >
> >
> > Exception Trace:
> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_0, Status : FAILED
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_1, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> > either because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> > at $Proxy7.create(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> > at $Proxy7.create(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> > at
> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> > at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> > at
> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> > at
> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_2, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
> either
> > because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> >
> >
> > Thanks & Regards,
> > B Anil Kumar.
>
>
>
> --
> Harsh J
>
Re: Job fails while re attempting the task in multiple outputs case
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Harsh.
@Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
I am using Apache Hadoop's multipleOutputs.
But as you see in stack trace, it's not appending the attempt id to file
name, it's only consists of task id.
Thanks & Regards,
B Anil Kumar.
On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
> Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
>
> If its the latter, please take a look at gotchas to take care of
> described at
> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>
> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
> wrote:
> > Hi,
> >
> > I am using multiple outputs in our job. So whenever any reduce task
> fails,
> > all it's next task attempts are failing with file exist exception.
> >
> >
> > The output file name should also append the task attempt right? But it's
> > only appending the task id. Is this the bug or Some thing wrong from my
> > side?
> >
> > Where should look in src code? I went through code at
> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
> task
> > id.
> >
> >
> > Exception Trace:
> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_0, Status : FAILED
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_1, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> > either because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> > at $Proxy7.create(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> > at $Proxy7.create(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> > at
> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> > at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> > at
> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> > at
> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_2, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
> either
> > because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> >
> >
> > Thanks & Regards,
> > B Anil Kumar.
>
>
>
> --
> Harsh J
>
Re: Job fails while re attempting the task in multiple outputs case
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Harsh.
@Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
I am using Apache Hadoop's multipleOutputs.
But as you see in stack trace, it's not appending the attempt id to file
name, it's only consists of task id.
Thanks & Regards,
B Anil Kumar.
On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
> Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
>
> If its the latter, please take a look at gotchas to take care of
> described at
> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>
> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
> wrote:
> > Hi,
> >
> > I am using multiple outputs in our job. So whenever any reduce task
> fails,
> > all it's next task attempts are failing with file exist exception.
> >
> >
> > The output file name should also append the task attempt right? But it's
> > only appending the task id. Is this the bug or Some thing wrong from my
> > side?
> >
> > Where should look in src code? I went through code at
> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
> task
> > id.
> >
> >
> > Exception Trace:
> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_0, Status : FAILED
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_1, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> > either because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> > at $Proxy7.create(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> > at $Proxy7.create(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> > at
> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> > at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> > at
> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> > at
> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_2, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
> either
> > because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> >
> >
> > Thanks & Regards,
> > B Anil Kumar.
>
>
>
> --
> Harsh J
>
Re: Job fails while re attempting the task in multiple outputs case
Posted by AnilKumar B <ak...@gmail.com>.
Thanks Harsh.
@Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
I am using Apache Hadoop's multipleOutputs.
But as you see in stack trace, it's not appending the attempt id to file
name, it's only consists of task id.
Thanks & Regards,
B Anil Kumar.
On Mon, Dec 30, 2013 at 7:42 PM, Harsh J <ha...@cloudera.com> wrote:
> Are you using the MultipleOutputs class shipped with Apache Hadoop or
> one of your own?
>
> If its the latter, please take a look at gotchas to take care of
> described at
> http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
>
> On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com>
> wrote:
> > Hi,
> >
> > I am using multiple outputs in our job. So whenever any reduce task
> fails,
> > all it's next task attempts are failing with file exist exception.
> >
> >
> > The output file name should also append the task attempt right? But it's
> > only appending the task id. Is this the bug or Some thing wrong from my
> > side?
> >
> > Where should look in src code? I went through code at
> > FileOutputFormat$getTaskOutputPath(), but there it's only considering
> task
> > id.
> >
> >
> > Exception Trace:
> > 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_0, Status : FAILED
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> > 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> > 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> > 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_1, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> > either because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at
> > org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> > at $Proxy7.create(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> > at $Proxy7.create(Unknown Source)
> > at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> > at
> > org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> > at
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> > at
> > org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> > at
> >
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> > at
> > com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> > at
> org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> > at
> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> > at
> > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> > at java.security.AccessController.doPrivileged(Native
> > Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> > at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> > 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> >
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> > 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> > 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> > 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> > 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> > 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> > attempt_201312162255_60465_r_000008_2, Status : FAILED
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to
> create
> > /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33
> either
> > because the filename is invalid or the file exists
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> > at
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> > Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> > at
> > org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> >
> >
> > Thanks & Regards,
> > B Anil Kumar.
>
>
>
> --
> Harsh J
>
Re: Job fails while re attempting the task in multiple outputs case
Posted by Harsh J <ha...@cloudera.com>.
Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
If its the latter, please take a look at gotchas to take care of
described at http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com> wrote:
> Hi,
>
> I am using multiple outputs in our job. So whenever any reduce task fails,
> all it's next task attempts are failing with file exist exception.
>
>
> The output file name should also append the task attempt right? But it's
> only appending the task id. Is this the bug or Some thing wrong from my
> side?
>
> Where should look in src code? I went through code at
> FileOutputFormat$getTaskOutputPath(), but there it's only considering task
> id.
>
>
> Exception Trace:
> 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_0, Status : FAILED
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> either because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at $Proxy7.create(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at $Proxy7.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> at
> org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_2, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33 either
> because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>
>
> Thanks & Regards,
> B Anil Kumar.
--
Harsh J
Re: Job fails while re attempting the task in multiple outputs case
Posted by Harsh J <ha...@cloudera.com>.
Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
If its the latter, please take a look at gotchas to take care of
described at http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com> wrote:
> Hi,
>
> I am using multiple outputs in our job. So whenever any reduce task fails,
> all it's next task attempts are failing with file exist exception.
>
>
> The output file name should also append the task attempt right? But it's
> only appending the task id. Is this the bug or Some thing wrong from my
> side?
>
> Where should look in src code? I went through code at
> FileOutputFormat$getTaskOutputPath(), but there it's only considering task
> id.
>
>
> Exception Trace:
> 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_0, Status : FAILED
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> either because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at $Proxy7.create(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at $Proxy7.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> at
> org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_2, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33 either
> because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>
>
> Thanks & Regards,
> B Anil Kumar.
--
Harsh J
Re: Job fails while re attempting the task in multiple outputs case
Posted by Harsh J <ha...@cloudera.com>.
Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
If its the latter, please take a look at gotchas to take care of
described at http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com> wrote:
> Hi,
>
> I am using multiple outputs in our job. So whenever any reduce task fails,
> all it's next task attempts are failing with file exist exception.
>
>
> The output file name should also append the task attempt right? But it's
> only appending the task id. Is this the bug or Some thing wrong from my
> side?
>
> Where should look in src code? I went through code at
> FileOutputFormat$getTaskOutputPath(), but there it's only considering task
> id.
>
>
> Exception Trace:
> 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_0, Status : FAILED
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> either because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at $Proxy7.create(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at $Proxy7.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> at
> org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_2, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33 either
> because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>
>
> Thanks & Regards,
> B Anil Kumar.
--
Harsh J
Re: Job fails while re attempting the task in multiple outputs case
Posted by Harsh J <ha...@cloudera.com>.
Are you using the MultipleOutputs class shipped with Apache Hadoop or
one of your own?
If its the latter, please take a look at gotchas to take care of
described at http://wiki.apache.org/hadoop/FAQ#Can_I_write_create.2Fwrite-to_hdfs_files_directly_from_map.2Freduce_tasks.3F
On Mon, Dec 30, 2013 at 4:22 PM, AnilKumar B <ak...@gmail.com> wrote:
> Hi,
>
> I am using multiple outputs in our job. So whenever any reduce task fails,
> all it's next task attempts are failing with file exist exception.
>
>
> The output file name should also append the task attempt right? But it's
> only appending the task id. Is this the bug or Some thing wrong from my
> side?
>
> Where should look in src code? I went through code at
> FileOutputFormat$getTaskOutputPath(), but there it's only considering task
> id.
>
>
> Exception Trace:
> 13/12/29 09:13:00 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_0, Status : FAILED
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stdout
> 13/12/29 09:14:42 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_0&filter=stderr
> 13/12/29 09:15:04 INFO mapred.JobClient: map 100% reduce 93%
> 13/12/29 09:15:23 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:17:31 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:19:34 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_1, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.10.31
> either because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1442)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at $Proxy7.create(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at $Proxy7.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.<init>(DFSClient.java:3753)
> at
> org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:937)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:207)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
> at
> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
> at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:131)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.getRecordWriter(MultipleOutputs.java:411)
> at
> org.apache.hadoop.mapreduce.lib.output.MultipleOutputs.write(MultipleOutputs.java:370)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java:254)
> at
> com.team.hadoop.mapreduce1$Reducer1.reduce(MapReduce1.java::144)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)
> at
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
> at
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stdout
> 13/12/29 09:19:35 WARN mapred.JobClient: Error reading task
> outputhttp://localhost:50050/tasklog?plaintext=true&attemptid=attempt_201312162255_60465_r_000008_1&filter=stderr
> 13/12/29 09:19:36 INFO mapred.JobClient: map 100% reduce 89%
> 13/12/29 09:19:54 INFO mapred.JobClient: map 100% reduce 92%
> 13/12/29 09:20:11 INFO mapred.JobClient: map 100% reduce 96%
> 13/12/29 09:22:52 INFO mapred.JobClient: map 100% reduce 97%
> 13/12/29 09:23:48 INFO mapred.JobClient: Task Id :
> attempt_201312162255_60465_r_000008_2, Status : FAILED
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: failed to create
> /x/y/z/2013/12/29/04/o_2013_12_29_03-r-00008.gz on client 10.103.7.33 either
> because the filename is invalid or the file exists
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1672)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1599)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:732)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.create(NameNode.java:711)
> at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown
> Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
> at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1448)
>
>
> Thanks & Regards,
> B Anil Kumar.
--
Harsh J