Problem trying to append file

[Aaron Baff <Aa...@telescope.tv>]

I'm trying out the HDFS Append with r0.21.0 in a quick little simple application before using in my production code. It creates a new file, writes out 50K records, closes the file, then opens the file in append, writes another 100K records, then closes the file. Everything is fine up until it goes to close the append stream. When it does that, I get the below error. Furthermore, when I check the datanode logfile for one that had the issue, I'm seeing some stuff in the logfile relating to that block. First is the output from my small application, 2nd is the output from the datanode logfile.

I can't do a hadoop fs -fsck at this moment due to some production data being used (very small cluster, we're a pretty new Hadoop shop), so while I'm pretty sure an fsck would be fine, I want to wait until we're done with it before running that, or asking the sysadmin to restart the DFS. This will be early tomorrow at the earliest, but I can try just about any other suggestions. Help!

--Aaron

03-21-11 15:58:17 [INFO ] Exception in createBlockOutputStream java.io.EOFException
03-21-11 15:58:17 [WARN ] Error Recovery for block blk_8212105008236569520_123591 in pipeline 10.10.11.50:50010, 10.10.11.51:50010, 10.10.11.52:50010: bad datanode 10.10.11.50:50010
03-21-11 15:58:18 [INFO ] Exception in createBlockOutputStream java.io.EOFException
03-21-11 15:58:18 [WARN ] Error Recovery for block blk_8212105008236569520_123591 in pipeline 10.10.11.51:50010, 10.10.11.52:50010: bad datanode 10.10.11.51:50010
03-21-11 15:58:18 [INFO ] Exception in createBlockOutputStream java.io.EOFException
03-21-11 15:58:18 [WARN ] DataStreamer Exception: java.lang.NullPointerException
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:480)

03-21-11 15:58:18 [WARN ] DFSOutputStream ResponseProcessor exception  for block blk_8212105008236569520_123591java.lang.NullPointerException
            at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:531)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:616)

03-21-11 15:58:19 [ERROR] java.io.IOException - All datanodes 10.10.11.52:50010 are bad. Aborting...
java.io.IOException: All datanodes 10.10.11.52:50010 are bad. Aborting...
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:753)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:431)
03-21-11 15:58:19 [ERROR] Exception closing file /user/abaff/test/append/test4.out : java.io.IOException: All datanodes 10.10.11.52:50010 are bad. Aborting...
java.io.IOException: All datanodes 10.10.11.52:50010 are bad. Aborting...
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:753)
            at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:431)




2011-03-21 15:58:17,694 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_8212105008236569520_123591 src: /10.8.4.6:3812 dest: /10.10.11.50:50010
2011-03-21 15:58:17,695 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Appending to replica FinalizedReplica, blk_8212105008236569520_123591, FINALIZED
  getNumBytes()     = 1538890
  getBytesOnDisk()  = 1538890
  getVisibleLength()= 1538890
  getVolume()       = /hadoop/hadoop-datastore/hadoop-hadoop/dfs/data/current/finalized
  getBlockFile()    = /hadoop/hadoop-datastore/hadoop-hadoop/dfs/data/current/finalized/subdir14/subdir45/blk_8212105008236569520
  unlinked=false
2011-03-21 15:58:17,735 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver constructor. Cause is
2011-03-21 15:58:17,736 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock blk_8212105008236569520_123591 received exception java.io.IOException: Failed to get link count on file /hadoop/hadoop-datastore/hadoop-hadoop/dfs/data/current/finalized/subdir14/subdir45/blk_8212105008236569520: message=null; error=stat: illegal option -- c; exit value=1
2011-03-21 15:58:17,736 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(10.10.11.50:50010, storageID=DS-1360244006-10.10.11.50-50010-1290637962096, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Failed to get link count on file /hadoop/hadoop-datastore/hadoop-hadoop/dfs/data/current/finalized/subdir14/subdir45/blk_8212105008236569520: message=null; error=stat: illegal option -- c; exit value=1
            at org.apache.hadoop.fs.FileUtil.createIOException(FileUtil.java:709)
            at org.apache.hadoop.fs.FileUtil.access$000(FileUtil.java:42)
            at org.apache.hadoop.fs.FileUtil$HardLink.getLinkCount(FileUtil.java:682)
            at org.apache.hadoop.hdfs.server.datanode.ReplicaInfo.unlinkBlock(ReplicaInfo.java:215)
            at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1116)
            at org.apache.hadoop.hdfs.server.datanode.FSDataset.append(FSDataset.java:1099)
            at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:112)
            at org.apache.hadoop.hdfs.server.datanode.DataXceiver.opWriteBlock(DataXceiver.java:258)
            at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.opWriteBlock(DataTransferProtocol.java:390)
            at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Receiver.processOp(DataTransferProtocol.java:331)
            at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:111)
            at java.lang.Thread.run(Thread.java:619)