DFS # of blocks

[Michael Basnight <mb...@gmail.com>]

howdy all, im doing some hadoop testing (so im still new to it), and  
im running into an error. ( DataStreamer Exception:  
java.io.IOException: Unable to create new block.) My DFS is not large  
(971 files and directories, 3906 blocks = 4877 total. Heap Size is  
13.9 MB / 966.69 MB (1%) ), and definitely not full ;), and hadoop is  
writing to a separate file for each reducer output key (roughly 4K  
keys), but after like 900 it just dies. I am using a child of  
MultipleTextOutputFormat<Text, Text> for my output format that creates  
a file for the key and puts the list of values in it in the same dfs  
folder. Below is the last 20 lines of logs for the node that it failed  
on. Any thoughts?

eTask: Read 122146089 bytes from map-output for  
attempt_200905051459_0001_m_000000_0
2009-05-05 15:09:59,672 INFO org.apache.hadoop.mapred.ReduceTask: Rec  
#1 from attempt_200905051459_0001_m_000000_0 -> (19, 2771) from hadoop2
2009-05-05 15:10:00,652 INFO org.apache.hadoop.mapred.ReduceTask:  
GetMapEventsThread exiting
2009-05-05 15:10:00,652 INFO org.apache.hadoop.mapred.ReduceTask:  
getMapsEventsThread joined.
2009-05-05 15:10:00,652 INFO org.apache.hadoop.mapred.ReduceTask:  
Closed ram manager
2009-05-05 15:10:00,652 INFO org.apache.hadoop.mapred.ReduceTask:  
Interleaved on-disk merge complete: 0 files left.
2009-05-05 15:10:00,652 INFO org.apache.hadoop.mapred.ReduceTask: In- 
memory merge complete: 2 files left.
2009-05-05 15:10:00,780 INFO org.apache.hadoop.mapred.Merger: Merging  
2 sorted segments
2009-05-05 15:10:00,780 INFO org.apache.hadoop.mapred.Merger: Down to  
the last merge-pass, with 2 segments left of total size: 245292507 bytes
2009-05-05 15:10:06,276 INFO org.apache.hadoop.mapred.ReduceTask:  
Merged 2 segments, 245292507 bytes to disk to satisfy reduce memory  
limit
2009-05-05 15:10:06,280 INFO org.apache.hadoop.mapred.ReduceTask:  
Merging 1 files, 245292509 bytes from disk
2009-05-05 15:10:06,284 INFO org.apache.hadoop.mapred.ReduceTask:  
Merging 0 segments, 0 bytes from memory into reduce
2009-05-05 15:10:06,284 INFO org.apache.hadoop.mapred.Merger: Merging  
1 sorted segments
2009-05-05 15:10:06,312 INFO org.apache.hadoop.mapred.Merger: Down to  
the last merge-pass, with 1 segments left of total size: 245292505 bytes
2009-05-05 15:10:35,346 INFO org.apache.hadoop.hdfs.DFSClient:  
Exception in createBlockOutputStream java.io.EOFException
2009-05-05 15:10:35,374 INFO org.apache.hadoop.hdfs.DFSClient:  
Abandoning block blk_-2874778794594289753_8565
2009-05-05 15:10:41,402 INFO org.apache.hadoop.hdfs.DFSClient:  
Exception in createBlockOutputStream java.io.EOFException
2009-05-05 15:10:41,402 INFO org.apache.hadoop.hdfs.DFSClient:  
Abandoning block blk_243746846946054460_8565
2009-05-05 15:10:47,411 INFO org.apache.hadoop.hdfs.DFSClient:  
Exception in createBlockOutputStream java.io.EOFException
2009-05-05 15:10:47,411 INFO org.apache.hadoop.hdfs.DFSClient:  
Abandoning block blk_-4508716893999737242_8565
2009-05-05 15:10:53,419 INFO org.apache.hadoop.hdfs.DFSClient:  
Exception in createBlockOutputStream java.io.EOFException
2009-05-05 15:10:53,419 INFO org.apache.hadoop.hdfs.DFSClient:  
Abandoning block blk_-2679897353336358687_8565
2009-05-05 15:10:59,423 WARN org.apache.hadoop.hdfs.DFSClient:  
DataStreamer Exception: java.io.IOException: Unable to create new block.
	at org.apache.hadoop.hdfs.DFSClient 
$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2781)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access 
$2000(DFSClient.java:2046)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream 
$DataStreamer.run(DFSClient.java:2232)

2009-05-05 15:10:59,423 WARN org.apache.hadoop.hdfs.DFSClient: Error  
Recovery for block blk_-2679897353336358687_8565 bad datanode[0] nodes  
== null
2009-05-05 15:10:59,423 WARN org.apache.hadoop.hdfs.DFSClient: Could  
not get block locations. Source file "/testing/output/medium_output/ 
_temporary/_attempt_200905051459_0001_r_000000_3/sometestingkey" -  
Aborting...
2009-05-05 15:12:24,960 WARN org.apache.hadoop.mapred.TaskTracker:  
Error running child
java.io.EOFException
	at java.io.DataInputStream.readByte(Unknown Source)
	at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
	at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
	at org.apache.hadoop.io.Text.readString(Text.java:400)
	at org.apache.hadoop.hdfs.DFSClient 
$DFSOutputStream.createBlockOutputStream(DFSClient.java:2837)
	at org.apache.hadoop.hdfs.DFSClient 
$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2762)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access 
$2000(DFSClient.java:2046)
	at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream 
$DataStreamer.run(DFSClient.java:2232)
2009-05-05 15:12:24,972 INFO org.apache.hadoop.mapred.TaskRunner:  
Runnning cleanup for the task