You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Jason Venner <ja...@attributor.com> on 2007/12/04 00:44:20 UTC

dfs failures with 0.15.0 - blk_3105072074036734167 has already been started

on XX.YY.ZZ.AAA
2007-12-03 15:42:18,508 ERROR org.apache.hadoop.dfs.DataNode: 
DataXceiver: java.io.IOException: Block blk_3105072074036734167 has 
already been started (though not completed), and thus cannot be created.
        at org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:568)
        at 
org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1257)
        at 
org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:901)
        at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:804)
        at java.lang.Thread.run(Thread.java:619)

One one machine
2007-12-03 15:42:44,755 INFO org.apache.hadoop.dfs.DataNode: Starting 
thread to transfer block blk_3105072074036734167 to 
[Lorg.apache.hadoop.dfs.DatanodeInfo;@6fc40f
2007-12-03 15:42:44,757 WARN org.apache.hadoop.dfs.DataNode: Failed to 
transfer blk_3105072074036734167 to XX.YY.ZZ.AAA:50010 got 
java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at 
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at 
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at 
org.apache.hadoop.dfs.DataNode$BlockSender.sendChunk(DataNode.java:1175)
        at 
org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:1208)
        at 
org.apache.hadoop.dfs.DataNode$DataTransfer.run(DataNode.java:1460)
        at java.lang.Thread.run(Thread.java:619)

Re: dfs failures with 0.15.0 - blk_3105072074036734167 has already been started

Posted by Jason Venner <ja...@attributor.com>.

 From the web page monitor, the job appears to be successfully running. 
The only thing is that there is no progress.

07/12/03 15:04:27 INFO mapred.FileInputFormat: Total input paths to 
process : 789
07/12/03 15:04:28 INFO mapred.JobClient: Running job: job_200711301649_0018
07/12/03 15:04:29 INFO mapred.JobClient:  map 0% reduce 0%
07/12/03 15:04:37 INFO mapred.JobClient:  map 1% reduce 0%
07/12/03 15:04:42 INFO mapred.JobClient:  map 2% reduce 0%
07/12/03 15:04:49 INFO mapred.JobClient:  map 3% reduce 0%
07/12/03 15:04:52 INFO mapred.JobClient:  map 4% reduce 0%
07/12/03 15:04:59 INFO mapred.JobClient:  map 5% reduce 0%
07/12/03 15:05:05 INFO mapred.JobClient:  map 6% reduce 0%
07/12/03 15:05:12 INFO mapred.JobClient:  map 7% reduce 0%
07/12/03 15:05:16 INFO mapred.JobClient:  map 8% reduce 0%
07/12/03 15:05:23 INFO mapred.JobClient:  map 9% reduce 0%
07/12/03 15:05:28 INFO mapred.JobClient:  map 10% reduce 0%
07/12/03 15:05:35 INFO mapred.JobClient:  map 11% reduce 0%
07/12/03 15:05:40 INFO mapred.JobClient:  map 12% reduce 0%
07/12/03 15:05:46 INFO mapred.JobClient:  map 13% reduce 0%
07/12/03 15:05:51 INFO mapred.JobClient:  map 14% reduce 0%
07/12/03 15:05:58 INFO mapred.JobClient:  map 15% reduce 0%
07/12/03 15:06:04 INFO mapred.JobClient:  map 16% reduce 0%
07/12/03 15:06:11 INFO mapred.JobClient:  map 17% reduce 0%
07/12/03 15:06:15 INFO mapred.JobClient:  map 18% reduce 0%
07/12/03 15:06:22 INFO mapred.JobClient:  map 19% reduce 0%
07/12/03 15:06:29 INFO mapred.JobClient:  map 20% reduce 0%
07/12/03 15:06:36 INFO mapred.JobClient:  map 21% reduce 0%
07/12/03 15:06:42 INFO mapred.JobClient:  map 22% reduce 0%
07/12/03 15:06:48 INFO mapred.JobClient:  map 23% reduce 0%
07/12/03 15:06:55 INFO mapred.JobClient:  map 24% reduce 0%
07/12/03 15:07:00 INFO mapred.JobClient:  map 25% reduce 0%
07/12/03 15:07:08 INFO mapred.JobClient:  map 26% reduce 0%
07/12/03 15:07:12 INFO mapred.JobClient:  map 27% reduce 0%
07/12/03 15:07:19 INFO mapred.JobClient:  map 28% reduce 0%
07/12/03 15:07:24 INFO mapred.JobClient:  map 29% reduce 0%
07/12/03 15:07:31 INFO mapred.JobClient:  map 30% reduce 0%
07/12/03 15:07:37 INFO mapred.JobClient:  map 31% reduce 0%
07/12/03 15:07:42 INFO mapred.JobClient:  map 32% reduce 0%
07/12/03 15:07:50 INFO mapred.JobClient:  map 33% reduce 0%
07/12/03 15:07:54 INFO mapred.JobClient:  map 34% reduce 0%
07/12/03 15:08:01 INFO mapred.JobClient:  map 35% reduce 0%
07/12/03 15:08:07 INFO mapred.JobClient:  map 36% reduce 0%
07/12/03 15:08:13 INFO mapred.JobClient:  map 37% reduce 0%
07/12/03 15:08:17 INFO mapred.JobClient:  map 38% reduce 0%
07/12/03 15:08:19 INFO mapred.JobClient:  map 38% reduce 1%
07/12/03 15:08:24 INFO mapred.JobClient:  map 39% reduce 1%
07/12/03 15:08:29 INFO mapred.JobClient:  map 40% reduce 1%
07/12/03 15:08:36 INFO mapred.JobClient:  map 41% reduce 1%
07/12/03 15:08:43 INFO mapred.JobClient:  map 42% reduce 1%
07/12/03 15:08:47 INFO mapred.JobClient:  map 43% reduce 1%
07/12/03 15:08:56 INFO mapred.JobClient:  map 44% reduce 1%
07/12/03 15:09:02 INFO mapred.JobClient:  map 45% reduce 1%
07/12/03 15:09:09 INFO mapred.JobClient:  map 46% reduce 1%
07/12/03 15:09:18 INFO mapred.JobClient:  map 47% reduce 1%
07/12/03 15:09:25 INFO mapred.JobClient:  map 48% reduce 1%
07/12/03 15:09:36 INFO mapred.JobClient:  map 49% reduce 1%
07/12/03 15:09:44 INFO mapred.JobClient:  map 50% reduce 1%
07/12/03 15:09:52 INFO mapred.JobClient:  map 51% reduce 1%
07/12/03 15:09:57 INFO mapred.JobClient:  map 52% reduce 1%
07/12/03 15:10:06 INFO mapred.JobClient:  map 53% reduce 1%
07/12/03 15:10:13 INFO mapred.JobClient:  map 54% reduce 1%
07/12/03 15:10:20 INFO mapred.JobClient:  map 55% reduce 1%
07/12/03 15:10:30 INFO mapred.JobClient:  map 56% reduce 1%
07/12/03 15:10:35 INFO mapred.JobClient:  map 57% reduce 1%
07/12/03 15:10:42 INFO mapred.JobClient:  map 58% reduce 1%
07/12/03 15:10:52 INFO mapred.JobClient:  map 59% reduce 1%
07/12/03 15:10:55 INFO mapred.JobClient:  map 60% reduce 1%
07/12/03 15:11:04 INFO mapred.JobClient:  map 61% reduce 1%
07/12/03 15:11:11 INFO mapred.JobClient:  map 62% reduce 1%
07/12/03 15:11:17 INFO mapred.JobClient:  map 63% reduce 1%
07/12/03 15:11:26 INFO mapred.JobClient:  map 64% reduce 1%
07/12/03 15:11:30 INFO mapred.JobClient:  map 65% reduce 1%
07/12/03 15:11:34 INFO mapred.JobClient:  map 65% reduce 2%
07/12/03 15:11:39 INFO mapred.JobClient:  map 66% reduce 2%
07/12/03 15:11:45 INFO mapred.JobClient:  map 67% reduce 2%
07/12/03 15:11:52 INFO mapred.JobClient:  map 68% reduce 2%
07/12/03 15:12:01 INFO mapred.JobClient:  map 69% reduce 2%
07/12/03 15:12:06 INFO mapred.JobClient:  map 70% reduce 2%
07/12/03 15:12:13 INFO mapred.JobClient:  map 71% reduce 2%
07/12/03 15:12:21 INFO mapred.JobClient:  map 72% reduce 2%
07/12/03 15:12:25 INFO mapred.JobClient:  map 73% reduce 2%
07/12/03 15:12:34 INFO mapred.JobClient:  map 74% reduce 2%
07/12/03 15:12:41 INFO mapred.JobClient:  map 75% reduce 2%
07/12/03 15:12:47 INFO mapred.JobClient:  map 76% reduce 2%
07/12/03 15:12:56 INFO mapred.JobClient:  map 77% reduce 2%
07/12/03 15:13:01 INFO mapred.JobClient:  map 78% reduce 2%
07/12/03 15:13:07 INFO mapred.JobClient:  map 79% reduce 2%
07/12/03 15:13:14 INFO mapred.JobClient:  map 80% reduce 2%
07/12/03 15:13:20 INFO mapred.JobClient:  map 81% reduce 2%
07/12/03 15:13:25 INFO mapred.JobClient:  map 82% reduce 2%
07/12/03 15:13:30 INFO mapred.JobClient:  map 83% reduce 2%
07/12/03 15:13:38 INFO mapred.JobClient:  map 84% reduce 2%
07/12/03 15:13:42 INFO mapred.JobClient:  map 85% reduce 2%
07/12/03 15:13:49 INFO mapred.JobClient:  map 86% reduce 2%
07/12/03 15:13:53 INFO mapred.JobClient:  map 87% reduce 2%
07/12/03 15:14:02 INFO mapred.JobClient:  map 88% reduce 2%
07/12/03 15:14:05 INFO mapred.JobClient:  map 89% reduce 2%
07/12/03 15:14:14 INFO mapred.JobClient:  map 90% reduce 2%
07/12/03 15:14:15 INFO mapred.JobClient:  map 91% reduce 2%
07/12/03 15:14:24 INFO mapred.JobClient:  map 92% reduce 2%
07/12/03 15:14:27 INFO mapred.JobClient:  map 93% reduce 2%
07/12/03 15:14:36 INFO mapred.JobClient:  map 94% reduce 2%
07/12/03 15:14:39 INFO mapred.JobClient:  map 95% reduce 2%
07/12/03 15:14:44 INFO mapred.JobClient:  map 95% reduce 3%
07/12/03 15:14:48 INFO mapred.JobClient:  map 96% reduce 3%
07/12/03 15:14:58 INFO mapred.JobClient:  map 97% reduce 3%
07/12/03 15:15:07 INFO mapred.JobClient:  map 98% reduce 3%
07/12/03 15:15:19 INFO mapred.JobClient:  map 99% reduce 3%
- stalled here for 45 minutes now.

Jason Venner wrote:
> on XX.YY.ZZ.AAA
> 2007-12-03 15:42:18,508 ERROR org.apache.hadoop.dfs.DataNode: 
> DataXceiver: java.io.IOException: Block blk_3105072074036734167 has 
> already been started (though not completed), and thus cannot be created.
>        at 
> org.apache.hadoop.dfs.FSDataset.writeToBlock(FSDataset.java:568)
>        at 
> org.apache.hadoop.dfs.DataNode$BlockReceiver.<init>(DataNode.java:1257)
>        at 
> org.apache.hadoop.dfs.DataNode$DataXceiver.writeBlock(DataNode.java:901)
>        at 
> org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:804)
>        at java.lang.Thread.run(Thread.java:619)
>
> One one machine
> 2007-12-03 15:42:44,755 INFO org.apache.hadoop.dfs.DataNode: Starting 
> thread to transfer block blk_3105072074036734167 to 
> [Lorg.apache.hadoop.dfs.DatanodeInfo;@6fc40f
> 2007-12-03 15:42:44,757 WARN org.apache.hadoop.dfs.DataNode: Failed to 
> transfer blk_3105072074036734167 to XX.YY.ZZ.AAA:50010 got 
> java.net.SocketException: Broken pipe
>        at java.net.SocketOutputStream.socketWrite0(Native Method)
>        at 
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
>        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>        at 
> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>        at 
> java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
>        at java.io.DataOutputStream.write(DataOutputStream.java:90)
>        at 
> org.apache.hadoop.dfs.DataNode$BlockSender.sendChunk(DataNode.java:1175)
>        at 
> org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:1208)
>        at 
> org.apache.hadoop.dfs.DataNode$DataTransfer.run(DataNode.java:1460)
>        at java.lang.Thread.run(Thread.java:619)
>