randomtextwriter can not write 100GB text file on a 500GB cluster

[Richard Zhang <ri...@gmail.com>]

Hi folks:
I am trying to write a 100GB text file on a cluster with 500GB free storage
space. For smaller scale writing such as 100MB, 1G, it works fine. But it
shows the DFS client can not complete writing errors when I ran with 100GB
writing. Does anyone has any ideas on these types of errors or met it
before?

Below is the command I used:

bin/hadoop jar hadoop-0.17.0-examples.jar randomtextwriter -D
test.randomtextwrite.min_words_key=5 -D
test.randomtextwrite.max_words_key=10 -D
test.randomtextwrite.min_words_value=20
test.randomtextwrite.max_words_value=100  -D
test.randomtextwrite.total_bytes=107374182000 rand-text.
And the below are the errors:

08/06/29 14:18:18 INFO mapred.JobClient:  map 1% reduce 0%
08/06/29 14:18:20 INFO mapred.JobClient: Task Id :
task_200806291409_0001_m_000012_0, Status : FAILED
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Could not
complete write to file
/user/hadoop/rand-text/_temporary/_task_200806291409_0001_m_000012_0/part-00012
by DFSClient_task_200806291409_0001_m_000012_0
        at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:332)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)

        at org.apache.hadoop.ipc.Client.call(Client.java:557)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
        at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
        at org.apache.hadoop.dfs.$Proxy1.complete(Unknown Source)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:2655)
        at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.close(DFSClient.java:2576)
        at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:59)
        at
org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:79)
        at
org.apache.hadoop.io.SequenceFile$Writer.close(SequenceFile.java:969)
        at
org.apache.hadoop.mapred.SequenceFileOutputFormat$1.close(SequenceFileOutputFormat.java:79)
        at
org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.close(MapTask.java:257)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:224)
        at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)