You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Johnson Chen <do...@gmail.com> on 2009/07/02 10:47:05 UTC
Block not found
Hi ,
My hadoop program stop at 66% Reduce phrase
09/07/02 16:41:37 INFO mapred.JobClient: map 0% reduce 0%
09/07/02 16:41:48 INFO mapred.JobClient: map 50% reduce 0%
09/07/02 16:41:51 INFO mapred.JobClient: map 100% reduce 0%
09/07/02 16:41:58 INFO mapred.JobClient: map 100% reduce 66%
And I found it pop up a lot of error message in namenode log .
here is the error messages :
2009-07-02 16:43:44,461 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
commitBlockSynchronization(lastblock=blk_-7800669485846603924_144754,
newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false,
deleteBlock=true)
2009-07-02 16:43:44,461 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 50040, call
commitBlockSynchronization(blk_-7800669485846603924_144754, 0, 0, false,
true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@2df2888) from
192.168.151.231:39976: error: java.io.IOException: Block
(=blk_-7800669485846603924_144754) not found
java.io.IOException: Block (=blk_-7800669485846603924_144754) not found
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:1906)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronization(NameNode.java:410)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
2009-07-02 16:43:44,967 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
commitBlockSynchronization(lastblock=blk_-8293900342000823254_748370,
newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false,
deleteBlock=true)
2009-07-02 16:43:44,967 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50040, call
commitBlockSynchronization(blk_-8293900342000823254_748370, 0, 0, false,
true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@8ddfa31) from
192.168.151.232:45283: error: java.io.IOException: Block
(=blk_-8293900342000823254_748370) not found
java.io.IOException: Block (=blk_-8293900342000823254_748370) not found
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:1906)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronization(NameNode.java:410)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
Can anyone help me ? Thanks.
--
Best wishes,
Johnson Chen
Re: Block not found
Posted by Bill Graham <bi...@gmail.com>.
I ran into the same issue when using the default settings for dfs.data.dir,
which is under the /tmp directory. Files in this directory will be cleaned
our periodically as needed by the OS, which will break HDFS.
On Thu, Jul 2, 2009 at 7:01 AM, Gross, Danny <Da...@spansion.com>wrote:
> Hello Johnson,
>
> I have observed similar error messages when my system ran out of disk
> space on an HDFS node, or in hadoop.tmp.dir.
>
> Hope it helps.
>
> Best regards,
>
> Danny
>
> -----Original Message-----
> From: Johnson Chen [mailto:dongogo@gmail.com]
> Sent: Thursday, July 02, 2009 3:47 AM
> To: core-user@hadoop.apache.org
> Subject: Block not found
>
> Hi ,
>
> My hadoop program stop at 66% Reduce phrase
>
> 09/07/02 16:41:37 INFO mapred.JobClient: map 0% reduce 0%
> 09/07/02 16:41:48 INFO mapred.JobClient: map 50% reduce 0%
> 09/07/02 16:41:51 INFO mapred.JobClient: map 100% reduce 0%
> 09/07/02 16:41:58 INFO mapred.JobClient: map 100% reduce 66%
>
> And I found it pop up a lot of error message in namenode log .
>
> here is the error messages :
>
> 2009-07-02 16:43:44,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> commitBlockSynchronization(lastblock=blk_-7800669485846603924_144754,
> newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false,
> deleteBlock=true)
> 2009-07-02 16:43:44,461 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 3 on 50040, call
> commitBlockSynchronization(blk_-7800669485846603924_144754, 0, 0, false,
> true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@2df2888) from
> 192.168.151.231:39976: error: java.io.IOException: Block
> (=blk_-7800669485846603924_144754) not found
> java.io.IOException: Block (=blk_-7800669485846603924_144754) not found
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchroni
> zation(FSNamesystem.java:1906)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronizati
> on(NameNode.java:410)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
> 2009-07-02 16:43:44,967 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> commitBlockSynchronization(lastblock=blk_-8293900342000823254_748370,
> newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false,
> deleteBlock=true)
> 2009-07-02 16:43:44,967 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 0 on 50040, call
> commitBlockSynchronization(blk_-8293900342000823254_748370, 0, 0, false,
> true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@8ddfa31) from
> 192.168.151.232:45283: error: java.io.IOException: Block
> (=blk_-8293900342000823254_748370) not found
> java.io.IOException: Block (=blk_-8293900342000823254_748370) not found
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchroni
> zation(FSNamesystem.java:1906)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronizati
> on(NameNode.java:410)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
> Impl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
>
>
> Can anyone help me ? Thanks.
>
> --
> Best wishes,
> Johnson Chen
>
RE: Block not found
Posted by "Gross, Danny" <Da...@spansion.com>.
Hello Johnson,
I have observed similar error messages when my system ran out of disk
space on an HDFS node, or in hadoop.tmp.dir.
Hope it helps.
Best regards,
Danny
-----Original Message-----
From: Johnson Chen [mailto:dongogo@gmail.com]
Sent: Thursday, July 02, 2009 3:47 AM
To: core-user@hadoop.apache.org
Subject: Block not found
Hi ,
My hadoop program stop at 66% Reduce phrase
09/07/02 16:41:37 INFO mapred.JobClient: map 0% reduce 0%
09/07/02 16:41:48 INFO mapred.JobClient: map 50% reduce 0%
09/07/02 16:41:51 INFO mapred.JobClient: map 100% reduce 0%
09/07/02 16:41:58 INFO mapred.JobClient: map 100% reduce 66%
And I found it pop up a lot of error message in namenode log .
here is the error messages :
2009-07-02 16:43:44,461 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
commitBlockSynchronization(lastblock=blk_-7800669485846603924_144754,
newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false,
deleteBlock=true)
2009-07-02 16:43:44,461 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 50040, call
commitBlockSynchronization(blk_-7800669485846603924_144754, 0, 0, false,
true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@2df2888) from
192.168.151.231:39976: error: java.io.IOException: Block
(=blk_-7800669485846603924_144754) not found
java.io.IOException: Block (=blk_-7800669485846603924_144754) not found
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchroni
zation(FSNamesystem.java:1906)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronizati
on(NameNode.java:410)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
2009-07-02 16:43:44,967 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
commitBlockSynchronization(lastblock=blk_-8293900342000823254_748370,
newgenerationstamp=0, newlength=0, newtargets=[], closeFile=false,
deleteBlock=true)
2009-07-02 16:43:44,967 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 50040, call
commitBlockSynchronization(blk_-8293900342000823254_748370, 0, 0, false,
true, [Lorg.apache.hadoop.hdfs.protocol.DatanodeID;@8ddfa31) from
192.168.151.232:45283: error: java.io.IOException: Block
(=blk_-8293900342000823254_748370) not found
java.io.IOException: Block (=blk_-8293900342000823254_748370) not found
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchroni
zation(FSNamesystem.java:1906)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.commitBlockSynchronizati
on(NameNode.java:410)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)
Can anyone help me ? Thanks.
--
Best wishes,
Johnson Chen