You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Vimal Jain <vk...@gmail.com> on 2013/08/13 10:23:37 UTC
Exceptions in Name node and Data node logs
Hi,
I have configured Hadoop and Hbase in pseudo distributed mode.
So far things were working fine , but suddenly i started receiving some
exceptions in my namenode and datanode log files.
It keeps repeating and thus fills up my disk space.
Please help here.
*Exception in data node :-*
2013-07-31 19:39:51,094 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
eived message from unregistered or dead node blk_-4787262105551508952_28369
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
at org.apache.hadoop.ipc.Client.call(Client.java:1107)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at sun.proxy.$Proxy5.blockReceived(Unknown Source)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
at java.lang.Thread.run(Thread.java:662)
*Exception in name node :- *
2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
dead
or unregistered node 192.168.20.30:50010
2013-07-31 19:39:50,671 ERROR
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hadoop cause:java.io.IOException: Got blo
ckReceived message from unregistered or dead node
blk_-4787262105551508952_28369
2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 9000, call blockReceived(DatanodeRegistration(
192.168.20.30:50010,
storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException:
Got blockReceived message from unregistered or dead node blk_-4787262105551
508952_28369
java.io.IOException: Got blockReceived message from unregistered or dead
node blk_-4787262105551508952_28369
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by t578384 <ha...@gmail.com>.
Hi Vimal ,
Could you elaborate this , what do you mean running 6 process in on single node,
How many nodes do you have in total and all are used for hadoop and hbase as well.
Possibilities: -
Might be there is dn no listed in your slave file and configure as DN in you cluster.
chances are your Nodes and not configure properly
thanks
Vikas Srivastava
On Aug 15, 2013, at 10:37 PM, Vimal Jain <vk...@gmail.com> wrote:
> Can someone please help here ?
>
>
> On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <je...@gmail.com> wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat message from DN but NN still getting block information from dead node. This error is similar to a bug HDFS-1250 reported 2 years back and fixed in 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> Exception in data node :-
>
> 2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> Exception in name node :-
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call blockReceived(DatanodeRegistration(192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075, ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by t578384 <ha...@gmail.com>.
Hi Vimal ,
Could you elaborate this , what do you mean running 6 process in on single node,
How many nodes do you have in total and all are used for hadoop and hbase as well.
Possibilities: -
Might be there is dn no listed in your slave file and configure as DN in you cluster.
chances are your Nodes and not configure properly
thanks
Vikas Srivastava
On Aug 15, 2013, at 10:37 PM, Vimal Jain <vk...@gmail.com> wrote:
> Can someone please help here ?
>
>
> On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <je...@gmail.com> wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat message from DN but NN still getting block information from dead node. This error is similar to a bug HDFS-1250 reported 2 years back and fixed in 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> Exception in data node :-
>
> 2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> Exception in name node :-
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call blockReceived(DatanodeRegistration(192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075, ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by t578384 <ha...@gmail.com>.
Hi Vimal ,
Could you elaborate this , what do you mean running 6 process in on single node,
How many nodes do you have in total and all are used for hadoop and hbase as well.
Possibilities: -
Might be there is dn no listed in your slave file and configure as DN in you cluster.
chances are your Nodes and not configure properly
thanks
Vikas Srivastava
On Aug 15, 2013, at 10:37 PM, Vimal Jain <vk...@gmail.com> wrote:
> Can someone please help here ?
>
>
> On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <je...@gmail.com> wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat message from DN but NN still getting block information from dead node. This error is similar to a bug HDFS-1250 reported 2 years back and fixed in 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> Exception in data node :-
>
> 2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> Exception in name node :-
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call blockReceived(DatanodeRegistration(192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075, ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by t578384 <ha...@gmail.com>.
Hi Vimal ,
Could you elaborate this , what do you mean running 6 process in on single node,
How many nodes do you have in total and all are used for hadoop and hbase as well.
Possibilities: -
Might be there is dn no listed in your slave file and configure as DN in you cluster.
chances are your Nodes and not configure properly
thanks
Vikas Srivastava
On Aug 15, 2013, at 10:37 PM, Vimal Jain <vk...@gmail.com> wrote:
> Can someone please help here ?
>
>
> On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <je...@gmail.com> wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat message from DN but NN still getting block information from dead node. This error is similar to a bug HDFS-1250 reported 2 years back and fixed in 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> Exception in data node :-
>
> 2013-07-31 19:39:51,094 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> Exception in name node :-
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9000, call blockReceived(DatanodeRegistration(192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075, ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error: java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead node blk_-4787262105551508952_28369
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Can someone please help here ?
On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi Jitendra,
>> Thanks for your reply.
>> Currently my hadoop/hbase is down in production as it had filled up the
>> disk space with above exceptions in log files and had to be brought down.
>> Also i am using hadoop/hbase in pseudo distributed mode , so there is
>> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>>
>>
>>
>> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
>> jeetuyadav200890@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> One of your DN is marked as dead because NN is not able to get heartbeat
>>> message from DN but NN still getting block information from dead node. This
>>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed
>>> in 0.20 release.
>>>
>>> Can you please check the status of DN's in cluster.
>>>
>>> #bin/hadoop dfsadmin -report
>>>
>>> Thanks
>>>
>>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>>> So far things were working fine , but suddenly i started receiving some
>>>> exceptions in my namenode and datanode log files.
>>>> It keeps repeating and thus fills up my disk space.
>>>> Please help here.
>>>>
>>>> *Exception in data node :-*
>>>>
>>>> 2013-07-31 19:39:51,094 WARN
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>>> eived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> *Exception in name node :- *
>>>>
>>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>>> dead
>>>> or unregistered node 192.168.20.30:50010
>>>> 2013-07-31 19:39:50,671 ERROR
>>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>>> as:hadoop cause:java.io.IOException: Got blo
>>>> ckReceived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>>> 192.168.20.30:50010,
>>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>>> infoPort=50075, ipcPort=50020),
>>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>>> node blk_-4787262105551
>>>> 508952_28369
>>>> java.io.IOException: Got blockReceived message from unregistered or
>>>> dead node blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Vimal Jain
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Can someone please help here ?
On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi Jitendra,
>> Thanks for your reply.
>> Currently my hadoop/hbase is down in production as it had filled up the
>> disk space with above exceptions in log files and had to be brought down.
>> Also i am using hadoop/hbase in pseudo distributed mode , so there is
>> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>>
>>
>>
>> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
>> jeetuyadav200890@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> One of your DN is marked as dead because NN is not able to get heartbeat
>>> message from DN but NN still getting block information from dead node. This
>>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed
>>> in 0.20 release.
>>>
>>> Can you please check the status of DN's in cluster.
>>>
>>> #bin/hadoop dfsadmin -report
>>>
>>> Thanks
>>>
>>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>>> So far things were working fine , but suddenly i started receiving some
>>>> exceptions in my namenode and datanode log files.
>>>> It keeps repeating and thus fills up my disk space.
>>>> Please help here.
>>>>
>>>> *Exception in data node :-*
>>>>
>>>> 2013-07-31 19:39:51,094 WARN
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>>> eived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> *Exception in name node :- *
>>>>
>>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>>> dead
>>>> or unregistered node 192.168.20.30:50010
>>>> 2013-07-31 19:39:50,671 ERROR
>>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>>> as:hadoop cause:java.io.IOException: Got blo
>>>> ckReceived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>>> 192.168.20.30:50010,
>>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>>> infoPort=50075, ipcPort=50020),
>>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>>> node blk_-4787262105551
>>>> 508952_28369
>>>> java.io.IOException: Got blockReceived message from unregistered or
>>>> dead node blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Vimal Jain
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Can someone please help here ?
On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi Jitendra,
>> Thanks for your reply.
>> Currently my hadoop/hbase is down in production as it had filled up the
>> disk space with above exceptions in log files and had to be brought down.
>> Also i am using hadoop/hbase in pseudo distributed mode , so there is
>> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>>
>>
>>
>> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
>> jeetuyadav200890@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> One of your DN is marked as dead because NN is not able to get heartbeat
>>> message from DN but NN still getting block information from dead node. This
>>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed
>>> in 0.20 release.
>>>
>>> Can you please check the status of DN's in cluster.
>>>
>>> #bin/hadoop dfsadmin -report
>>>
>>> Thanks
>>>
>>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>>> So far things were working fine , but suddenly i started receiving some
>>>> exceptions in my namenode and datanode log files.
>>>> It keeps repeating and thus fills up my disk space.
>>>> Please help here.
>>>>
>>>> *Exception in data node :-*
>>>>
>>>> 2013-07-31 19:39:51,094 WARN
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>>> eived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> *Exception in name node :- *
>>>>
>>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>>> dead
>>>> or unregistered node 192.168.20.30:50010
>>>> 2013-07-31 19:39:50,671 ERROR
>>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>>> as:hadoop cause:java.io.IOException: Got blo
>>>> ckReceived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>>> 192.168.20.30:50010,
>>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>>> infoPort=50075, ipcPort=50020),
>>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>>> node blk_-4787262105551
>>>> 508952_28369
>>>> java.io.IOException: Got blockReceived message from unregistered or
>>>> dead node blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Vimal Jain
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Can someone please help here ?
On Tue, Aug 13, 2013 at 9:28 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> As Jitendra pointed out , this issue was fixed in .20 version.
> I am using Hadoop 1.1.2 so why its occurring again ?
> Please help here.
>
>
> On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi Jitendra,
>> Thanks for your reply.
>> Currently my hadoop/hbase is down in production as it had filled up the
>> disk space with above exceptions in log files and had to be brought down.
>> Also i am using hadoop/hbase in pseudo distributed mode , so there is
>> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>>
>>
>>
>> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
>> jeetuyadav200890@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> One of your DN is marked as dead because NN is not able to get heartbeat
>>> message from DN but NN still getting block information from dead node. This
>>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed
>>> in 0.20 release.
>>>
>>> Can you please check the status of DN's in cluster.
>>>
>>> #bin/hadoop dfsadmin -report
>>>
>>> Thanks
>>>
>>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>>> So far things were working fine , but suddenly i started receiving some
>>>> exceptions in my namenode and datanode log files.
>>>> It keeps repeating and thus fills up my disk space.
>>>> Please help here.
>>>>
>>>> *Exception in data node :-*
>>>>
>>>> 2013-07-31 19:39:51,094 WARN
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>>> eived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>>> at
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>>
>>>> *Exception in name node :- *
>>>>
>>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>>> dead
>>>> or unregistered node 192.168.20.30:50010
>>>> 2013-07-31 19:39:50,671 ERROR
>>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>>> as:hadoop cause:java.io.IOException: Got blo
>>>> ckReceived message from unregistered or dead node
>>>> blk_-4787262105551508952_28369
>>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>>> 192.168.20.30:50010,
>>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>>> infoPort=50075, ipcPort=50020),
>>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>>> node blk_-4787262105551
>>>> 508952_28369
>>>> java.io.IOException: Got blockReceived message from unregistered or
>>>> dead node blk_-4787262105551508952_28369
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>>> at
>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>> Vimal Jain
>>>>
>>>
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi,
As Jitendra pointed out , this issue was fixed in .20 version.
I am using Hadoop 1.1.2 so why its occurring again ?
Please help here.
On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the
> disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is
> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
> jeetuyadav200890@gmail.com> wrote:
>
>> Hi,
>>
>> One of your DN is marked as dead because NN is not able to get heartbeat
>> message from DN but NN still getting block information from dead node. This
>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
>> 0.20 release.
>>
>> Can you please check the status of DN's in cluster.
>>
>> #bin/hadoop dfsadmin -report
>>
>> Thanks
>>
>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>
>>> Hi,
>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>> So far things were working fine , but suddenly i started receiving some
>>> exceptions in my namenode and datanode log files.
>>> It keeps repeating and thus fills up my disk space.
>>> Please help here.
>>>
>>> *Exception in data node :-*
>>>
>>> 2013-07-31 19:39:51,094 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>> eived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> *Exception in name node :- *
>>>
>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>> dead
>>> or unregistered node 192.168.20.30:50010
>>> 2013-07-31 19:39:50,671 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:hadoop cause:java.io.IOException: Got blo
>>> ckReceived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>> 192.168.20.30:50010,
>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>> infoPort=50075, ipcPort=50020),
>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551
>>> 508952_28369
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Vimal Jain
>>>
>>
>>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi,
As Jitendra pointed out , this issue was fixed in .20 version.
I am using Hadoop 1.1.2 so why its occurring again ?
Please help here.
On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the
> disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is
> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
> jeetuyadav200890@gmail.com> wrote:
>
>> Hi,
>>
>> One of your DN is marked as dead because NN is not able to get heartbeat
>> message from DN but NN still getting block information from dead node. This
>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
>> 0.20 release.
>>
>> Can you please check the status of DN's in cluster.
>>
>> #bin/hadoop dfsadmin -report
>>
>> Thanks
>>
>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>
>>> Hi,
>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>> So far things were working fine , but suddenly i started receiving some
>>> exceptions in my namenode and datanode log files.
>>> It keeps repeating and thus fills up my disk space.
>>> Please help here.
>>>
>>> *Exception in data node :-*
>>>
>>> 2013-07-31 19:39:51,094 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>> eived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> *Exception in name node :- *
>>>
>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>> dead
>>> or unregistered node 192.168.20.30:50010
>>> 2013-07-31 19:39:50,671 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:hadoop cause:java.io.IOException: Got blo
>>> ckReceived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>> 192.168.20.30:50010,
>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>> infoPort=50075, ipcPort=50020),
>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551
>>> 508952_28369
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Vimal Jain
>>>
>>
>>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi,
As Jitendra pointed out , this issue was fixed in .20 version.
I am using Hadoop 1.1.2 so why its occurring again ?
Please help here.
On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the
> disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is
> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
> jeetuyadav200890@gmail.com> wrote:
>
>> Hi,
>>
>> One of your DN is marked as dead because NN is not able to get heartbeat
>> message from DN but NN still getting block information from dead node. This
>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
>> 0.20 release.
>>
>> Can you please check the status of DN's in cluster.
>>
>> #bin/hadoop dfsadmin -report
>>
>> Thanks
>>
>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>
>>> Hi,
>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>> So far things were working fine , but suddenly i started receiving some
>>> exceptions in my namenode and datanode log files.
>>> It keeps repeating and thus fills up my disk space.
>>> Please help here.
>>>
>>> *Exception in data node :-*
>>>
>>> 2013-07-31 19:39:51,094 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>> eived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> *Exception in name node :- *
>>>
>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>> dead
>>> or unregistered node 192.168.20.30:50010
>>> 2013-07-31 19:39:50,671 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:hadoop cause:java.io.IOException: Got blo
>>> ckReceived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>> 192.168.20.30:50010,
>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>> infoPort=50075, ipcPort=50020),
>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551
>>> 508952_28369
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Vimal Jain
>>>
>>
>>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi,
As Jitendra pointed out , this issue was fixed in .20 version.
I am using Hadoop 1.1.2 so why its occurring again ?
Please help here.
On Tue, Aug 13, 2013 at 2:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi Jitendra,
> Thanks for your reply.
> Currently my hadoop/hbase is down in production as it had filled up the
> disk space with above exceptions in log files and had to be brought down.
> Also i am using hadoop/hbase in pseudo distributed mode , so there is
> only one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
>
>
>
> On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav <
> jeetuyadav200890@gmail.com> wrote:
>
>> Hi,
>>
>> One of your DN is marked as dead because NN is not able to get heartbeat
>> message from DN but NN still getting block information from dead node. This
>> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
>> 0.20 release.
>>
>> Can you please check the status of DN's in cluster.
>>
>> #bin/hadoop dfsadmin -report
>>
>> Thanks
>>
>> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>>
>>> Hi,
>>> I have configured Hadoop and Hbase in pseudo distributed mode.
>>> So far things were working fine , but suddenly i started receiving some
>>> exceptions in my namenode and datanode log files.
>>> It keeps repeating and thus fills up my disk space.
>>> Please help here.
>>>
>>> *Exception in data node :-*
>>>
>>> 2013-07-31 19:39:51,094 WARN
>>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>>> eived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>>> at java.lang.Thread.run(Thread.java:662)
>>>
>>> *Exception in name node :- *
>>>
>>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>>> dead
>>> or unregistered node 192.168.20.30:50010
>>> 2013-07-31 19:39:50,671 ERROR
>>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>>> as:hadoop cause:java.io.IOException: Got blo
>>> ckReceived message from unregistered or dead node
>>> blk_-4787262105551508952_28369
>>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>>> 192.168.20.30:50010,
>>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>>> infoPort=50075, ipcPort=50020),
>>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551
>>> 508952_28369
>>> java.io.IOException: Got blockReceived message from unregistered or dead
>>> node blk_-4787262105551508952_28369
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>>> at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>>
>>>
>>> --
>>> Thanks and Regards,
>>> Vimal Jain
>>>
>>
>>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi Jitendra,
Thanks for your reply.
Currently my hadoop/hbase is down in production as it had filled up the
disk space with above exceptions in log files and had to be brought down.
Also i am using hadoop/hbase in pseudo distributed mode , so there is only
one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat
> message from DN but NN still getting block information from dead node. This
> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
> 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi Jitendra,
Thanks for your reply.
Currently my hadoop/hbase is down in production as it had filled up the
disk space with above exceptions in log files and had to be brought down.
Also i am using hadoop/hbase in pseudo distributed mode , so there is only
one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat
> message from DN but NN still getting block information from dead node. This
> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
> 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi Jitendra,
Thanks for your reply.
Currently my hadoop/hbase is down in production as it had filled up the
disk space with above exceptions in log files and had to be brought down.
Also i am using hadoop/hbase in pseudo distributed mode , so there is only
one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat
> message from DN but NN still getting block information from dead node. This
> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
> 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Hi Jitendra,
Thanks for your reply.
Currently my hadoop/hbase is down in production as it had filled up the
disk space with above exceptions in log files and had to be brought down.
Also i am using hadoop/hbase in pseudo distributed mode , so there is only
one node which hosts all 6 processes ( 3 from hadoop and 3 from hbase).
On Tue, Aug 13, 2013 at 2:50 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Hi,
>
> One of your DN is marked as dead because NN is not able to get heartbeat
> message from DN but NN still getting block information from dead node. This
> error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
> 0.20 release.
>
> Can you please check the status of DN's in cluster.
>
> #bin/hadoop dfsadmin -report
>
> Thanks
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
One of your DN is marked as dead because NN is not able to get heartbeat
message from DN but NN still getting block information from dead node. This
error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
0.20 release.
Can you please check the status of DN's in cluster.
#bin/hadoop dfsadmin -report
Thanks
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
Re: Exceptions in Name node and Data node logs
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
One of your DN is marked as dead because NN is not able to get heartbeat
message from DN but NN still getting block information from dead node. This
error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
0.20 release.
Can you please check the status of DN's in cluster.
#bin/hadoop dfsadmin -report
Thanks
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
Re: Exceptions in Name node and Data node logs
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
One of your DN is marked as dead because NN is not able to get heartbeat
message from DN but NN still getting block information from dead node. This
error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
0.20 release.
Can you please check the status of DN's in cluster.
#bin/hadoop dfsadmin -report
Thanks
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
Re: Exceptions in Name node and Data node logs
Posted by Jitendra Yadav <je...@gmail.com>.
Hi,
One of your DN is marked as dead because NN is not able to get heartbeat
message from DN but NN still getting block information from dead node. This
error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
0.20 release.
Can you please check the status of DN's in cluster.
#bin/hadoop dfsadmin -report
Thanks
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Along with these exceptions i am seeing some exceptions in hbase logs too.
Here it is :
*Exception in Master log :*
2013-07-31 15:51:04,694 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
1266874891120ms instead of 10000ms, this is likely due to a long garbage c
ollecting pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 15:51:04,798 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
Failed scan of catalog table
org.apache.hadoop.hbase.client.ScannerTimeoutException: 82253ms passed
since the last invocation, timeout is currently set to 60000
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:283)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:727)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:184)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:169)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:123)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:134)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:92)
at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: -8839286818925700393
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2544)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:143)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:274)
... 8 more
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2013-07-31 15:54:42,531 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY =>
'', ENCODED => 1028785192,}
2013-07-31 15:54:42,532 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 5 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2013-07-31 15:54:42,751 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=5 averag
e=5.0 mostloaded=5 leastloaded=5
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
25771ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
30091ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
28613ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
27457ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,362 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
34587ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,367 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78600ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,369 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78562ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
*Exception in Region log :
*
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1375)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1319)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1480)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1271)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
... 5 more
Caused by: java.io.IOException: DFSOutputStream is closed
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3669)
at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:995)
... 9 more
2013-07-31 15:50:37,761 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
close of hlog
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
On Tue, Aug 13, 2013 at 1:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Sorry for not giving version details
> I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
>
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Along with these exceptions i am seeing some exceptions in hbase logs too.
Here it is :
*Exception in Master log :*
2013-07-31 15:51:04,694 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
1266874891120ms instead of 10000ms, this is likely due to a long garbage c
ollecting pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 15:51:04,798 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
Failed scan of catalog table
org.apache.hadoop.hbase.client.ScannerTimeoutException: 82253ms passed
since the last invocation, timeout is currently set to 60000
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:283)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:727)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:184)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:169)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:123)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:134)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:92)
at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: -8839286818925700393
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2544)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:143)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:274)
... 8 more
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2013-07-31 15:54:42,531 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY =>
'', ENCODED => 1028785192,}
2013-07-31 15:54:42,532 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 5 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2013-07-31 15:54:42,751 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=5 averag
e=5.0 mostloaded=5 leastloaded=5
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
25771ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
30091ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
28613ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
27457ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,362 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
34587ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,367 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78600ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,369 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78562ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
*Exception in Region log :
*
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1375)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1319)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1480)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1271)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
... 5 more
Caused by: java.io.IOException: DFSOutputStream is closed
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3669)
at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:995)
... 9 more
2013-07-31 15:50:37,761 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
close of hlog
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
On Tue, Aug 13, 2013 at 1:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Sorry for not giving version details
> I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
>
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Along with these exceptions i am seeing some exceptions in hbase logs too.
Here it is :
*Exception in Master log :*
2013-07-31 15:51:04,694 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
1266874891120ms instead of 10000ms, this is likely due to a long garbage c
ollecting pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 15:51:04,798 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
Failed scan of catalog table
org.apache.hadoop.hbase.client.ScannerTimeoutException: 82253ms passed
since the last invocation, timeout is currently set to 60000
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:283)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:727)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:184)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:169)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:123)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:134)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:92)
at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: -8839286818925700393
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2544)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:143)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:274)
... 8 more
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2013-07-31 15:54:42,531 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY =>
'', ENCODED => 1028785192,}
2013-07-31 15:54:42,532 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 5 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2013-07-31 15:54:42,751 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=5 averag
e=5.0 mostloaded=5 leastloaded=5
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
25771ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
30091ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
28613ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
27457ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,362 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
34587ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,367 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78600ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,369 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78562ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
*Exception in Region log :
*
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1375)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1319)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1480)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1271)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
... 5 more
Caused by: java.io.IOException: DFSOutputStream is closed
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3669)
at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:995)
... 9 more
2013-07-31 15:50:37,761 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
close of hlog
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
On Tue, Aug 13, 2013 at 1:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Sorry for not giving version details
> I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
>
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Along with these exceptions i am seeing some exceptions in hbase logs too.
Here it is :
*Exception in Master log :*
2013-07-31 15:51:04,694 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
1266874891120ms instead of 10000ms, this is likely due to a long garbage c
ollecting pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 15:51:04,798 WARN org.apache.hadoop.hbase.master.CatalogJanitor:
Failed scan of catalog table
org.apache.hadoop.hbase.client.ScannerTimeoutException: 82253ms passed
since the last invocation, timeout is currently set to 60000
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:283)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:727)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:184)
at
org.apache.hadoop.hbase.catalog.MetaReader.fullScan(MetaReader.java:169)
at
org.apache.hadoop.hbase.master.CatalogJanitor.getSplitParents(CatalogJanitor.java:123)
at
org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:134)
at
org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:92)
at org.apache.hadoop.hbase.Chore.run(Chore.java:67)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: Name: -8839286818925700393
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2544)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:143)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:42)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:164)
at
org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:274)
... 8 more
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2013-07-31 15:54:42,526 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2013-07-31 15:54:42,531 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY =>
'', ENCODED => 1028785192,}
2013-07-31 15:54:42,532 DEBUG
org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 5 catalog row(s) and
gc'd 0 unreferenced parent region(s)
2013-07-31 15:54:42,751 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing because balanced cluster; servers=1 regions=5 averag
e=5.0 mostloaded=5 leastloaded=5
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
25771ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,358 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
30091ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
28613ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,361 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
27457ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,362 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
34587ms instead of 10000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,367 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78600ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-07-31 16:43:23,369 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
78562ms instead of 60000ms, this is likely due to a long garbage collectin
g pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
*Exception in Region log :
*
java.io.IOException: Reflection
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:304)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1375)
at org.apache.hadoop.hbase.regionserver.wal.HLog.syncer(HLog.java:1319)
at org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:1480)
at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:1271)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(SequenceFileLogWriter.java:302)
... 5 more
Caused by: java.io.IOException: DFSOutputStream is closed
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3669)
at
org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
at
org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:995)
... 9 more
2013-07-31 15:50:37,761 FATAL
org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting
close of hlog
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
2013-07-31 19:23:38,289 INFO org.apache.hadoop.hdfs.DFSClient: Could not
complete file /hbase/.logs/hbase.xyz.com,60020,1370497170634/hbase.xyz.com%2C60020%2C1370497170634.1375265949987
retrying...
On Tue, Aug 13, 2013 at 1:56 PM, Vimal Jain <vk...@gmail.com> wrote:
> Sorry for not giving version details
> I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
>
>
> On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
>
>> Hi,
>> I have configured Hadoop and Hbase in pseudo distributed mode.
>> So far things were working fine , but suddenly i started receiving some
>> exceptions in my namenode and datanode log files.
>> It keeps repeating and thus fills up my disk space.
>> Please help here.
>>
>> *Exception in data node :-*
>>
>> 2013-07-31 19:39:51,094 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
>> eived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *Exception in name node :- *
>>
>> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
>> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
>> dead
>> or unregistered node 192.168.20.30:50010
>> 2013-07-31 19:39:50,671 ERROR
>> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
>> as:hadoop cause:java.io.IOException: Got blo
>> ckReceived message from unregistered or dead node
>> blk_-4787262105551508952_28369
>> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
>> handler 5 on 9000, call blockReceived(DatanodeRegistration(
>> 192.168.20.30:50010,
>> storageID=DS-1816106352-192.168.20.30-50010-1369314076237,
>> infoPort=50075, ipcPort=50020),
>> [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559, [Ljava.l
>> ang.String;@294f9d6) from 192.168.20.30:59764: error:
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551
>> 508952_28369
>> java.io.IOException: Got blockReceived message from unregistered or dead
>> node blk_-4787262105551508952_28369
>> at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>> at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>>
>>
>> --
>> Thanks and Regards,
>> Vimal Jain
>>
>
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Sorry for not giving version details
I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Sorry for not giving version details
I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Sorry for not giving version details
I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain
Re: Exceptions in Name node and Data node logs
Posted by Vimal Jain <vk...@gmail.com>.
Sorry for not giving version details
I am using Hadoop version - 1.1.2 and Hbase version - 0.94.7
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <vk...@gmail.com> wrote:
> Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at sun.proxy.$Proxy5.blockReceived(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
> at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
> or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
> at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>
> --
> Thanks and Regards,
> Vimal Jain
>
--
Thanks and Regards,
Vimal Jain