You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by prabhu K <pr...@gmail.com> on 2012/08/08 09:49:42 UTC
Data node error
Hi Users,
I have formatted hadoop cluster, formatted successfully. after stop &start
hadoop, i hit the jps command in master, getting fine, but in slave machine
am not getting data node, while see the data node log file, i am getting
following error.
Data node(slave1):
2012-08-08 00:16:44,033 WARN
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
Verification failed for blk_-3831635302961953167_1690. Exception : java.io.
IOException: Block blk_-3831635302961953167_1690 is not valid.
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
at
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
at
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
at
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
at java.lang.Thread.run(Thread.java:662)
*data node(slave2)*
2012-08-08 13:03:50,195 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
spaceID = 1434906924; datanode namespaceID = 474761520
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
Please help me on this issue.
Thanks,
Prabhu.
Re: Data node error
Posted by Sandeep Reddy P <sa...@gmail.com>.
Hi,
Just edit the namespace id in datanode. In your case it'll be in the
dfs.data.dir
vi /app/hadoop/tmp/dfs/data/current/VERSION
Replace namespace id 474761520(datanode namespace id) with 1434906924
(namenode name space id)
On Wed, Aug 8, 2012 at 4:15 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> For incompatible namespaceid error, there are two solutions available
> here. ****
>
> ** **
>
>
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> ****
>
> ** **
>
> I ran into this same issue and second solution fixed it. ****
>
> ** **
>
> ** **
> ------------------------------
>
> *From:* anil gupta [mailto:anilgupta84@gmail.com]
> *Sent:* Wednesday, August 08, 2012 1:36 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Data node error****
>
> ** **
>
> This link might provide you some more information:
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
>
> HTH,
> Anil****
>
> On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> ****
>
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil****
>
> ** **
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> ****
>
> Hi Users,****
>
> ****
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.****
>
> ****
>
> Data node(slave1):****
>
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)****
>
>
> *data node(slave2)*****
>
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)**
> **
>
> ****
>
> Please help me on this issue.****
>
> ****
>
> Thanks,****
>
> Prabhu.****
>
>
>
> ****
>
> --
> Thanks & Regards,
> Anil Gupta****
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta****
>
--
Thanks,
sandeep
Re: Data node error
Posted by Sandeep Reddy P <sa...@gmail.com>.
Hi,
Just edit the namespace id in datanode. In your case it'll be in the
dfs.data.dir
vi /app/hadoop/tmp/dfs/data/current/VERSION
Replace namespace id 474761520(datanode namespace id) with 1434906924
(namenode name space id)
On Wed, Aug 8, 2012 at 4:15 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> For incompatible namespaceid error, there are two solutions available
> here. ****
>
> ** **
>
>
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> ****
>
> ** **
>
> I ran into this same issue and second solution fixed it. ****
>
> ** **
>
> ** **
> ------------------------------
>
> *From:* anil gupta [mailto:anilgupta84@gmail.com]
> *Sent:* Wednesday, August 08, 2012 1:36 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Data node error****
>
> ** **
>
> This link might provide you some more information:
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
>
> HTH,
> Anil****
>
> On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> ****
>
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil****
>
> ** **
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> ****
>
> Hi Users,****
>
> ****
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.****
>
> ****
>
> Data node(slave1):****
>
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)****
>
>
> *data node(slave2)*****
>
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)**
> **
>
> ****
>
> Please help me on this issue.****
>
> ****
>
> Thanks,****
>
> Prabhu.****
>
>
>
> ****
>
> --
> Thanks & Regards,
> Anil Gupta****
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta****
>
--
Thanks,
sandeep
Re: Data node error
Posted by Sandeep Reddy P <sa...@gmail.com>.
Hi,
Just edit the namespace id in datanode. In your case it'll be in the
dfs.data.dir
vi /app/hadoop/tmp/dfs/data/current/VERSION
Replace namespace id 474761520(datanode namespace id) with 1434906924
(namenode name space id)
On Wed, Aug 8, 2012 at 4:15 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> For incompatible namespaceid error, there are two solutions available
> here. ****
>
> ** **
>
>
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> ****
>
> ** **
>
> I ran into this same issue and second solution fixed it. ****
>
> ** **
>
> ** **
> ------------------------------
>
> *From:* anil gupta [mailto:anilgupta84@gmail.com]
> *Sent:* Wednesday, August 08, 2012 1:36 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Data node error****
>
> ** **
>
> This link might provide you some more information:
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
>
> HTH,
> Anil****
>
> On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> ****
>
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil****
>
> ** **
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> ****
>
> Hi Users,****
>
> ****
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.****
>
> ****
>
> Data node(slave1):****
>
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)****
>
>
> *data node(slave2)*****
>
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)**
> **
>
> ****
>
> Please help me on this issue.****
>
> ****
>
> Thanks,****
>
> Prabhu.****
>
>
>
> ****
>
> --
> Thanks & Regards,
> Anil Gupta****
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta****
>
--
Thanks,
sandeep
Re: Data node error
Posted by Sandeep Reddy P <sa...@gmail.com>.
Hi,
Just edit the namespace id in datanode. In your case it'll be in the
dfs.data.dir
vi /app/hadoop/tmp/dfs/data/current/VERSION
Replace namespace id 474761520(datanode namespace id) with 1434906924
(namenode name space id)
On Wed, Aug 8, 2012 at 4:15 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:
> For incompatible namespaceid error, there are two solutions available
> here. ****
>
> ** **
>
>
> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
> ****
>
> ** **
>
> I ran into this same issue and second solution fixed it. ****
>
> ** **
>
> ** **
> ------------------------------
>
> *From:* anil gupta [mailto:anilgupta84@gmail.com]
> *Sent:* Wednesday, August 08, 2012 1:36 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Data node error****
>
> ** **
>
> This link might provide you some more information:
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
>
> HTH,
> Anil****
>
> On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> ****
>
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil****
>
> ** **
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> ****
>
> Hi Users,****
>
> ****
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.****
>
> ****
>
> Data node(slave1):****
>
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)****
>
>
> *data node(slave2)*****
>
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)**
> **
>
> ****
>
> Please help me on this issue.****
>
> ****
>
> Thanks,****
>
> Prabhu.****
>
>
>
> ****
>
> --
> Thanks & Regards,
> Anil Gupta****
>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta****
>
--
Thanks,
sandeep
RE: Data node error
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
For incompatible namespaceid error, there are two solutions available here.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
I ran into this same issue and second solution fixed it.
________________________________
From: anil gupta [mailto:anilgupta84@gmail.com]
Sent: Wednesday, August 08, 2012 1:36 PM
To: user@hadoop.apache.org
Subject: Re: Data node error
This link might provide you some more information: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com>> wrote:
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the data directories of Datanodes needs to be cleaned up. As far as i remember it's the directory which you mention in dfs.data.dir in hdfs-site.xml file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com>> wrote:
Hi Users,
I have formatted hadoop cluster, formatted successfully. after stop &start hadoop, i hit the jps command in master, getting fine, but in slave machine am not getting data node, while see the data node log file, i am getting following error.
Data node(slave1):
2012-08-08 00:16:44,033 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second Verification failed for blk_-3831635302961953167_1690. Exception : java.io<http://java.io>.
IOException: Block blk_-3831635302961953167_1690 is not valid.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
at java.lang.Thread.run(Thread.java:662)
data node(slave2)
2012-08-08 13:03:50,195 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
spaceID = 1434906924; datanode namespaceID = 474761520
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
Please help me on this issue.
Thanks,
Prabhu.
--
Thanks & Regards,
Anil Gupta
--
Thanks & Regards,
Anil Gupta
RE: Data node error
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
For incompatible namespaceid error, there are two solutions available here.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
I ran into this same issue and second solution fixed it.
________________________________
From: anil gupta [mailto:anilgupta84@gmail.com]
Sent: Wednesday, August 08, 2012 1:36 PM
To: user@hadoop.apache.org
Subject: Re: Data node error
This link might provide you some more information: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com>> wrote:
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the data directories of Datanodes needs to be cleaned up. As far as i remember it's the directory which you mention in dfs.data.dir in hdfs-site.xml file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com>> wrote:
Hi Users,
I have formatted hadoop cluster, formatted successfully. after stop &start hadoop, i hit the jps command in master, getting fine, but in slave machine am not getting data node, while see the data node log file, i am getting following error.
Data node(slave1):
2012-08-08 00:16:44,033 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second Verification failed for blk_-3831635302961953167_1690. Exception : java.io<http://java.io>.
IOException: Block blk_-3831635302961953167_1690 is not valid.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
at java.lang.Thread.run(Thread.java:662)
data node(slave2)
2012-08-08 13:03:50,195 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
spaceID = 1434906924; datanode namespaceID = 474761520
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
Please help me on this issue.
Thanks,
Prabhu.
--
Thanks & Regards,
Anil Gupta
--
Thanks & Regards,
Anil Gupta
RE: Data node error
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
For incompatible namespaceid error, there are two solutions available here.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
I ran into this same issue and second solution fixed it.
________________________________
From: anil gupta [mailto:anilgupta84@gmail.com]
Sent: Wednesday, August 08, 2012 1:36 PM
To: user@hadoop.apache.org
Subject: Re: Data node error
This link might provide you some more information: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com>> wrote:
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the data directories of Datanodes needs to be cleaned up. As far as i remember it's the directory which you mention in dfs.data.dir in hdfs-site.xml file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com>> wrote:
Hi Users,
I have formatted hadoop cluster, formatted successfully. after stop &start hadoop, i hit the jps command in master, getting fine, but in slave machine am not getting data node, while see the data node log file, i am getting following error.
Data node(slave1):
2012-08-08 00:16:44,033 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second Verification failed for blk_-3831635302961953167_1690. Exception : java.io<http://java.io>.
IOException: Block blk_-3831635302961953167_1690 is not valid.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
at java.lang.Thread.run(Thread.java:662)
data node(slave2)
2012-08-08 13:03:50,195 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
spaceID = 1434906924; datanode namespaceID = 474761520
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
Please help me on this issue.
Thanks,
Prabhu.
--
Thanks & Regards,
Anil Gupta
--
Thanks & Regards,
Anil Gupta
RE: Data node error
Posted by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com>.
For incompatible namespaceid error, there are two solutions available here.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
I ran into this same issue and second solution fixed it.
________________________________
From: anil gupta [mailto:anilgupta84@gmail.com]
Sent: Wednesday, August 08, 2012 1:36 PM
To: user@hadoop.apache.org
Subject: Re: Data node error
This link might provide you some more information: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com>> wrote:
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the data directories of Datanodes needs to be cleaned up. As far as i remember it's the directory which you mention in dfs.data.dir in hdfs-site.xml file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com>> wrote:
Hi Users,
I have formatted hadoop cluster, formatted successfully. after stop &start hadoop, i hit the jps command in master, getting fine, but in slave machine am not getting data node, while see the data node log file, i am getting following error.
Data node(slave1):
2012-08-08 00:16:44,033 WARN org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second Verification failed for blk_-3831635302961953167_1690. Exception : java.io<http://java.io>.
IOException: Block blk_-3831635302961953167_1690 is not valid.
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
at org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
at java.lang.Thread.run(Thread.java:662)
data node(slave2)
2012-08-08 13:03:50,195 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
spaceID = 1434906924; datanode namespaceID = 474761520
at org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
Please help me on this issue.
Thanks,
Prabhu.
--
Thanks & Regards,
Anil Gupta
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
This link might provide you some more information:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil
>
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
>
>> Hi Users,
>>
>> I have formatted hadoop cluster, formatted successfully. after stop
>> &start hadoop, i hit the jps command in master, getting fine, but in slave
>> machine am not getting data node, while see the data node log file, i am
>> getting following error.
>>
>> Data node(slave1):
>> 2012-08-08 00:16:44,033 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
>> Verification failed for blk_-3831635302961953167_1690. Exception :
>> java.io.
>> IOException: Block blk_-3831635302961953167_1690 is not valid.
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *data node(slave2)*
>> 2012-08-08 13:03:50,195 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
>> spaceID = 1434906924; datanode namespaceID = 474761520
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> Please help me on this issue.
>>
>> Thanks,
>> Prabhu.
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
This link might provide you some more information:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil
>
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
>
>> Hi Users,
>>
>> I have formatted hadoop cluster, formatted successfully. after stop
>> &start hadoop, i hit the jps command in master, getting fine, but in slave
>> machine am not getting data node, while see the data node log file, i am
>> getting following error.
>>
>> Data node(slave1):
>> 2012-08-08 00:16:44,033 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
>> Verification failed for blk_-3831635302961953167_1690. Exception :
>> java.io.
>> IOException: Block blk_-3831635302961953167_1690 is not valid.
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *data node(slave2)*
>> 2012-08-08 13:03:50,195 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
>> spaceID = 1434906924; datanode namespaceID = 474761520
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> Please help me on this issue.
>>
>> Thanks,
>> Prabhu.
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
This link might provide you some more information:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil
>
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
>
>> Hi Users,
>>
>> I have formatted hadoop cluster, formatted successfully. after stop
>> &start hadoop, i hit the jps command in master, getting fine, but in slave
>> machine am not getting data node, while see the data node log file, i am
>> getting following error.
>>
>> Data node(slave1):
>> 2012-08-08 00:16:44,033 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
>> Verification failed for blk_-3831635302961953167_1690. Exception :
>> java.io.
>> IOException: Block blk_-3831635302961953167_1690 is not valid.
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *data node(slave2)*
>> 2012-08-08 13:03:50,195 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
>> spaceID = 1434906924; datanode namespaceID = 474761520
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> Please help me on this issue.
>>
>> Thanks,
>> Prabhu.
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
This link might provide you some more information:
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201203.mbox/%3CCAAu13zHcMeUThvA9OPN9MeMyvE9SHxsD1gSdZnZOWs3qrQZaNQ@mail.gmail.com%3E
HTH,
Anil
On Wed, Aug 8, 2012 at 12:56 AM, anil gupta <an...@gmail.com> wrote:
> Hi Prabhu,
>
> Did you clean the data dir on DataNodes? Whenever Namenode is formated the
> data directories of Datanodes needs to be cleaned up. As far as i remember
> it's the directory which you mention in dfs.data.dir in hdfs-site.xml
> file.You can do a google search for the error and you can get more details.
> (Sorry, i dont have access to my cluster conf right now for telling you
> the exact property).
>
> Thanks,
> Anil
>
>
> On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
>
>> Hi Users,
>>
>> I have formatted hadoop cluster, formatted successfully. after stop
>> &start hadoop, i hit the jps command in master, getting fine, but in slave
>> machine am not getting data node, while see the data node log file, i am
>> getting following error.
>>
>> Data node(slave1):
>> 2012-08-08 00:16:44,033 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
>> Verification failed for blk_-3831635302961953167_1690. Exception :
>> java.io.
>> IOException: Block blk_-3831635302961953167_1690 is not valid.
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
>> at
>> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
>> at
>> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
>> at java.lang.Thread.run(Thread.java:662)
>>
>> *data node(slave2)*
>> 2012-08-08 13:03:50,195 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
>> spaceID = 1434906924; datanode namespaceID = 474761520
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> Please help me on this issue.
>>
>> Thanks,
>> Prabhu.
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the
data directories of Datanodes needs to be cleaned up. As far as i remember
it's the directory which you mention in dfs.data.dir in hdfs-site.xml
file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the
exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> Hi Users,
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.
>
> Data node(slave1):
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)
>
> *data node(slave2)*
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> Please help me on this issue.
>
> Thanks,
> Prabhu.
>
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the
data directories of Datanodes needs to be cleaned up. As far as i remember
it's the directory which you mention in dfs.data.dir in hdfs-site.xml
file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the
exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> Hi Users,
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.
>
> Data node(slave1):
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)
>
> *data node(slave2)*
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> Please help me on this issue.
>
> Thanks,
> Prabhu.
>
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the
data directories of Datanodes needs to be cleaned up. As far as i remember
it's the directory which you mention in dfs.data.dir in hdfs-site.xml
file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the
exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> Hi Users,
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.
>
> Data node(slave1):
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)
>
> *data node(slave2)*
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> Please help me on this issue.
>
> Thanks,
> Prabhu.
>
--
Thanks & Regards,
Anil Gupta
Re: Data node error
Posted by anil gupta <an...@gmail.com>.
Hi Prabhu,
Did you clean the data dir on DataNodes? Whenever Namenode is formated the
data directories of Datanodes needs to be cleaned up. As far as i remember
it's the directory which you mention in dfs.data.dir in hdfs-site.xml
file.You can do a google search for the error and you can get more details.
(Sorry, i dont have access to my cluster conf right now for telling you the
exact property).
Thanks,
Anil
On Wed, Aug 8, 2012 at 12:49 AM, prabhu K <pr...@gmail.com> wrote:
> Hi Users,
>
> I have formatted hadoop cluster, formatted successfully. after stop &start
> hadoop, i hit the jps command in master, getting fine, but in slave machine
> am not getting data node, while see the data node log file, i am getting
> following error.
>
> Data node(slave1):
> 2012-08-08 00:16:44,033 WARN
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Second
> Verification failed for blk_-3831635302961953167_1690. Exception : java.io
> .
> IOException: Block blk_-3831635302961953167_1690 is not valid.
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:1072)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:1035)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:1045)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:81)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyBlock(DataBlockScanner.java:453)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:617)
> at java.lang.Thread.run(Thread.java:662)
>
> *data node(slave2)*
> 2012-08-08 13:03:50,195 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode name
> spaceID = 1434906924; datanode namespaceID = 474761520
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> Please help me on this issue.
>
> Thanks,
> Prabhu.
>
--
Thanks & Regards,
Anil Gupta