You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Cyril Bogus <cy...@gmail.com> on 2013/03/13 15:27:07 UTC
Second node hdfs
I am trying to start the datanode on the slave node but when I check the
dfs I only have one node.
When I check the logs on the slave node I find the following output.
2013-03-13 10:22:14,608 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = Owner-5/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
2013-03-13 10:22:15,086 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2013-03-13 10:22:15,121 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
MetricsSystem,sub=Stats registered.
2013-03-13 10:22:15,123 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2013-03-13 10:22:15,123 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
started
2013-03-13 10:22:15,662 INFO
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
registered.
2013-03-13 10:22:15,686 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
exists!
2013-03-13 10:22:19,730 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
1683708441; datanode namespaceID = 606666501
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
2013-03-13 10:22:19,731 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
************************************************************/
Thank you for any insights.
Cyril
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
was this namenode part of any other hadoop cluster?
did you format your namenode and forgot to clean up the datanode?
you can refer Michael's blog for more details and solutions
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#javaioioexception-incompatible-namespaceids
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
--
Nitin Pawar
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
was this namenode part of any other hadoop cluster?
did you format your namenode and forgot to clean up the datanode?
you can refer Michael's blog for more details and solutions
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#javaioioexception-incompatible-namespaceids
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
--
Nitin Pawar
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
nvm. Thanks Nitin for the tutorial mentioning how to do this without
formatting the hdfs.
On Wed, Mar 13, 2013 at 11:43 AM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
you dont need to format each time you add a new node.
The previous node addition failed because the node you added had already
served a hadoop cluster and when you add it to a new one it showed
mismatch.
For a new node to add, just make sure that your datanode is clean (with the
data directory being newly created)
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
--
Nitin Pawar
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
you dont need to format each time you add a new node.
The previous node addition failed because the node you added had already
served a hadoop cluster and when you add it to a new one it showed
mismatch.
For a new node to add, just make sure that your datanode is clean (with the
data directory being newly created)
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
--
Nitin Pawar
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
No, you don't have to format the NN everytime you add a DN. Looking at your
case, it seems the second DN was part of some other cluster and has the ID
of that NN.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
No, you don't have to format the NN everytime you add a DN. Looking at your
case, it seems the second DN was part of some other cluster and has the ID
of that NN.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
No, you don't have to format the NN everytime you add a DN. Looking at your
case, it seems the second DN was part of some other cluster and has the ID
of that NN.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
you dont need to format each time you add a new node.
The previous node addition failed because the node you added had already
served a hadoop cluster and when you add it to a new one it showed
mismatch.
For a new node to add, just make sure that your datanode is clean (with the
data directory being newly created)
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
--
Nitin Pawar
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
nvm. Thanks Nitin for the tutorial mentioning how to do this without
formatting the hdfs.
On Wed, Mar 13, 2013 at 11:43 AM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
No, you don't have to format the NN everytime you add a DN. Looking at your
case, it seems the second DN was part of some other cluster and has the ID
of that NN.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
nvm. Thanks Nitin for the tutorial mentioning how to do this without
formatting the hdfs.
On Wed, Mar 13, 2013 at 11:43 AM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
you dont need to format each time you add a new node.
The previous node addition failed because the node you added had already
served a hadoop cluster and when you add it to a new one it showed
mismatch.
For a new node to add, just make sure that your datanode is clean (with the
data directory being newly created)
On Wed, Mar 13, 2013 at 9:13 PM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
--
Nitin Pawar
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
nvm. Thanks Nitin for the tutorial mentioning how to do this without
formatting the hdfs.
On Wed, Mar 13, 2013 at 11:43 AM, Cyril Bogus <cy...@gmail.com> wrote:
> Thank you both.
>
> So what both of you were saying which will be true is that is order to
> start and synchronize the cluster, I will have to format both nodes at the
> same time ok.
>
> I was working on the master node without the second node and did not
> format before trying to start the second one.
>
> I reformatted the cluster with both nodes connected and it worked. But I
> have a question.
>
> If I want to add a third node and my current cluster is populated with
> some tables, will I have to format it again in order to add the node?
>
>
> On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hello Cyril,
>>
>> This is because your datanode has a different namespaceID from the
>> one which master(namenode) actually has. Have you formatted the HDFS
>> recently? Were you able to format it properly? Everytime you format HDFS,
>> NameNode generates new namespaceID, which should be same in both NameNodes
>> and DataNodes otherwise it won't be able to reach NameNode.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com>wrote:
>>
>>> I am trying to start the datanode on the slave node but when I check the
>>> dfs I only have one node.
>>>
>>> When I check the logs on the slave node I find the following output.
>>>
>>> 2013-03-13 10:22:14,608 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>>> /************************************************************
>>> STARTUP_MSG: Starting DataNode
>>> STARTUP_MSG: host = Owner-5/127.0.1.1
>>> STARTUP_MSG: args = []
>>> STARTUP_MSG: version = 1.0.4
>>> STARTUP_MSG: build =
>>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>>> ************************************************************/
>>> 2013-03-13 10:22:15,086 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>>> hadoop-metrics2.properties
>>> 2013-03-13 10:22:15,121 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>>> MetricsSystem,sub=Stats registered.
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>>> period at 10 second(s).
>>> 2013-03-13 10:22:15,123 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>>> started
>>> 2013-03-13 10:22:15,662 INFO
>>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>>> registered.
>>> 2013-03-13 10:22:15,686 WARN
>>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>>> exists!
>>> 2013-03-13 10:22:19,730 ERROR
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>>> 1683708441; datanode namespaceID = 606666501
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>>> at
>>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>>
>>> 2013-03-13 10:22:19,731 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>>> ************************************************************/
>>>
>>> Thank you for any insights.
>>>
>>> Cyril
>>>
>>
>>
>
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
Thank you both.
So what both of you were saying which will be true is that is order to
start and synchronize the cluster, I will have to format both nodes at the
same time ok.
I was working on the master node without the second node and did not format
before trying to start the second one.
I reformatted the cluster with both nodes connected and it worked. But I
have a question.
If I want to add a third node and my current cluster is populated with some
tables, will I have to format it again in order to add the node?
On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Cyril,
>
> This is because your datanode has a different namespaceID from the
> one which master(namenode) actually has. Have you formatted the HDFS
> recently? Were you able to format it properly? Everytime you format HDFS,
> NameNode generates new namespaceID, which should be same in both NameNodes
> and DataNodes otherwise it won't be able to reach NameNode.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
>
>> I am trying to start the datanode on the slave node but when I check the
>> dfs I only have one node.
>>
>> When I check the logs on the slave node I find the following output.
>>
>> 2013-03-13 10:22:14,608 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = Owner-5/127.0.1.1
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2013-03-13 10:22:15,086 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-03-13 10:22:15,121 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>> started
>> 2013-03-13 10:22:15,662 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-03-13 10:22:15,686 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-03-13 10:22:19,730 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>> 1683708441; datanode namespaceID = 606666501
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> 2013-03-13 10:22:19,731 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>> ************************************************************/
>>
>> Thank you for any insights.
>>
>> Cyril
>>
>
>
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
Thank you both.
So what both of you were saying which will be true is that is order to
start and synchronize the cluster, I will have to format both nodes at the
same time ok.
I was working on the master node without the second node and did not format
before trying to start the second one.
I reformatted the cluster with both nodes connected and it worked. But I
have a question.
If I want to add a third node and my current cluster is populated with some
tables, will I have to format it again in order to add the node?
On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Cyril,
>
> This is because your datanode has a different namespaceID from the
> one which master(namenode) actually has. Have you formatted the HDFS
> recently? Were you able to format it properly? Everytime you format HDFS,
> NameNode generates new namespaceID, which should be same in both NameNodes
> and DataNodes otherwise it won't be able to reach NameNode.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
>
>> I am trying to start the datanode on the slave node but when I check the
>> dfs I only have one node.
>>
>> When I check the logs on the slave node I find the following output.
>>
>> 2013-03-13 10:22:14,608 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = Owner-5/127.0.1.1
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2013-03-13 10:22:15,086 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-03-13 10:22:15,121 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>> started
>> 2013-03-13 10:22:15,662 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-03-13 10:22:15,686 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-03-13 10:22:19,730 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>> 1683708441; datanode namespaceID = 606666501
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> 2013-03-13 10:22:19,731 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>> ************************************************************/
>>
>> Thank you for any insights.
>>
>> Cyril
>>
>
>
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
Thank you both.
So what both of you were saying which will be true is that is order to
start and synchronize the cluster, I will have to format both nodes at the
same time ok.
I was working on the master node without the second node and did not format
before trying to start the second one.
I reformatted the cluster with both nodes connected and it worked. But I
have a question.
If I want to add a third node and my current cluster is populated with some
tables, will I have to format it again in order to add the node?
On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Cyril,
>
> This is because your datanode has a different namespaceID from the
> one which master(namenode) actually has. Have you formatted the HDFS
> recently? Were you able to format it properly? Everytime you format HDFS,
> NameNode generates new namespaceID, which should be same in both NameNodes
> and DataNodes otherwise it won't be able to reach NameNode.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
>
>> I am trying to start the datanode on the slave node but when I check the
>> dfs I only have one node.
>>
>> When I check the logs on the slave node I find the following output.
>>
>> 2013-03-13 10:22:14,608 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = Owner-5/127.0.1.1
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2013-03-13 10:22:15,086 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-03-13 10:22:15,121 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>> started
>> 2013-03-13 10:22:15,662 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-03-13 10:22:15,686 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-03-13 10:22:19,730 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>> 1683708441; datanode namespaceID = 606666501
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> 2013-03-13 10:22:19,731 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>> ************************************************************/
>>
>> Thank you for any insights.
>>
>> Cyril
>>
>
>
Re: Second node hdfs
Posted by Cyril Bogus <cy...@gmail.com>.
Thank you both.
So what both of you were saying which will be true is that is order to
start and synchronize the cluster, I will have to format both nodes at the
same time ok.
I was working on the master node without the second node and did not format
before trying to start the second one.
I reformatted the cluster with both nodes connected and it worked. But I
have a question.
If I want to add a third node and my current cluster is populated with some
tables, will I have to format it again in order to add the node?
On Wed, Mar 13, 2013 at 10:34 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Cyril,
>
> This is because your datanode has a different namespaceID from the
> one which master(namenode) actually has. Have you formatted the HDFS
> recently? Were you able to format it properly? Everytime you format HDFS,
> NameNode generates new namespaceID, which should be same in both NameNodes
> and DataNodes otherwise it won't be able to reach NameNode.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
>
>> I am trying to start the datanode on the slave node but when I check the
>> dfs I only have one node.
>>
>> When I check the logs on the slave node I find the following output.
>>
>> 2013-03-13 10:22:14,608 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
>> /************************************************************
>> STARTUP_MSG: Starting DataNode
>> STARTUP_MSG: host = Owner-5/127.0.1.1
>> STARTUP_MSG: args = []
>> STARTUP_MSG: version = 1.0.4
>> STARTUP_MSG: build =
>> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
>> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
>> ************************************************************/
>> 2013-03-13 10:22:15,086 INFO
>> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
>> hadoop-metrics2.properties
>> 2013-03-13 10:22:15,121 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> MetricsSystem,sub=Stats registered.
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> period at 10 second(s).
>> 2013-03-13 10:22:15,123 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
>> started
>> 2013-03-13 10:22:15,662 INFO
>> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
>> registered.
>> 2013-03-13 10:22:15,686 WARN
>> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
>> exists!
>> 2013-03-13 10:22:19,730 ERROR
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
>> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
>> 1683708441; datanode namespaceID = 606666501
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
>> at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>>
>> 2013-03-13 10:22:19,731 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
>> /************************************************************
>> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
>> ************************************************************/
>>
>> Thank you for any insights.
>>
>> Cyril
>>
>
>
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Cyril,
This is because your datanode has a different namespaceID from the
one which master(namenode) actually has. Have you formatted the HDFS
recently? Were you able to format it properly? Everytime you format HDFS,
NameNode generates new namespaceID, which should be same in both NameNodes
and DataNodes otherwise it won't be able to reach NameNode.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
was this namenode part of any other hadoop cluster?
did you format your namenode and forgot to clean up the datanode?
you can refer Michael's blog for more details and solutions
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#javaioioexception-incompatible-namespaceids
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
--
Nitin Pawar
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Cyril,
This is because your datanode has a different namespaceID from the
one which master(namenode) actually has. Have you formatted the HDFS
recently? Were you able to format it properly? Everytime you format HDFS,
NameNode generates new namespaceID, which should be same in both NameNodes
and DataNodes otherwise it won't be able to reach NameNode.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Cyril,
This is because your datanode has a different namespaceID from the
one which master(namenode) actually has. Have you formatted the HDFS
recently? Were you able to format it properly? Everytime you format HDFS,
NameNode generates new namespaceID, which should be same in both NameNodes
and DataNodes otherwise it won't be able to reach NameNode.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
Re: Second node hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Cyril,
This is because your datanode has a different namespaceID from the
one which master(namenode) actually has. Have you formatted the HDFS
recently? Were you able to format it properly? Everytime you format HDFS,
NameNode generates new namespaceID, which should be same in both NameNodes
and DataNodes otherwise it won't be able to reach NameNode.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
Re: Second node hdfs
Posted by Nitin Pawar <ni...@gmail.com>.
was this namenode part of any other hadoop cluster?
did you format your namenode and forgot to clean up the datanode?
you can refer Michael's blog for more details and solutions
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/#javaioioexception-incompatible-namespaceids
On Wed, Mar 13, 2013 at 7:57 PM, Cyril Bogus <cy...@gmail.com> wrote:
> I am trying to start the datanode on the slave node but when I check the
> dfs I only have one node.
>
> When I check the logs on the slave node I find the following output.
>
> 2013-03-13 10:22:14,608 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting DataNode
> STARTUP_MSG: host = Owner-5/127.0.1.1
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 2013-03-13 10:22:15,086 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2013-03-13 10:22:15,121 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> MetricsSystem,sub=Stats registered.
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2013-03-13 10:22:15,123 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system
> started
> 2013-03-13 10:22:15,662 INFO
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> registered.
> 2013-03-13 10:22:15,686 WARN
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
> exists!
> 2013-03-13 10:22:19,730 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException:
> Incompatible namespaceIDs in /home/hadoop/hdfs/data: namenode namespaceID =
> 1683708441; datanode namespaceID = 606666501
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:232)
> at
> org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:147)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:385)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:299)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1582)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)
>
> 2013-03-13 10:22:19,731 INFO
> org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at Owner-5/127.0.1.1
> ************************************************************/
>
> Thank you for any insights.
>
> Cyril
>
--
Nitin Pawar