You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Colin Freas <co...@gmail.com> on 2008/03/21 18:39:58 UTC
Master as DataNode
setting up a simple hadoop cluster with two machines, i've gotten to the
point where the two machines can see each other, things seem fine, but i'm
trying to set up the master as both a master and a slave, just for testing
purposes.
so, i've put the master into the conf/masters file and the conf/slaves file.
things seem to work, but there's no DataNode process listed with jps on the
master. i'm wondering if there's a switch i need to flip to tell hadoop to
use the master as a datanode even if it's in the slaves file?
thanks again.
-colin
RE: Master as DataNode
Posted by Jeff Eastman <je...@windwardsolutions.com>.
I don't know the deep answer, but formatting your dfs creates a new
namespaceId that needs to be consistent across all slaves. Any data
directories containing old version ids will prevent the DataNode from
starting on that node. Maybe somebody who really knows the machinery can
elaborate to this.
Glad you are flying now,
Jeff
> -----Original Message-----
> From: Colin Freas [mailto:colinfreas@gmail.com]
> Sent: Friday, March 21, 2008 1:51 PM
> To: core-user@hadoop.apache.org
> Subject: Re: Master as DataNode
>
> yup, got it working with that technique.
>
> pushed it out to 5 machines, things look good. appreciate the help.
>
> what is it that causes this? i know i formatted the dfs more than once.
> is
> that what does it? or just adding nodes, or... ?
>
> -colin
>
>
> On Fri, Mar 21, 2008 at 2:30 PM, Jeff Eastman <je...@windwardsolutions.com>
> wrote:
>
> > I encountered this while I was starting out too, while moving from a
> > single
> > node cluster to more nodes. I suggest clearing your hadoop-datastore
> > directory, reformatting the HDFS and restarting again. You are very
> close
> > :)
> > Jeff
> >
> > > -----Original Message-----
> > > From: Colin Freas [mailto:colinfreas@gmail.com]
> > > Sent: Friday, March 21, 2008 11:18 AM
> > > To: core-user@hadoop.apache.org
> > > Subject: Re: Master as DataNode
> > >
> > > ah:
> > >
> > > 2008-03-21 14:06:05,526 ERROR org.apache.hadoop.dfs.DataNode:
> > > java.io.IOException: Incompatible namespaceIDs in
> > > /var/tmp/hadoop-datastore/hadoop/dfs/data: namenode namespaceID =
> > > 2121666262; datanode namespaceID = 2058961420
> > >
> > >
> > > looks like i'm hitting this "Incompatible namespaceID" bug:
> > > http://issues.apache.org/jira/browse/HADOOP-1212
> > >
> > > is there a work around for this?
> > >
> > > -colin
> > >
> > >
> > > On Fri, Mar 21, 2008 at 1:50 PM, Jeff Eastman <
> > jeff@windwardsolutions.com>
> > > wrote:
> > >
> > > > Check your logs. That should work out of the box with the
> > configuration
> > > > steps you described.
> > > >
> > > > Jeff
> > > >
> > > > > -----Original Message-----
> > > > > From: Colin Freas [mailto:colinfreas@gmail.com]
> > > > > Sent: Friday, March 21, 2008 10:40 AM
> > > > > To: core-user@hadoop.apache.org
> > > > > Subject: Master as DataNode
> > > > >
> > > > > setting up a simple hadoop cluster with two machines, i've gotten
> to
> > > the
> > > > > point where the two machines can see each other, things seem fine,
> > but
> > > > i'm
> > > > > trying to set up the master as both a master and a slave, just for
> > > > testing
> > > > > purposes.
> > > > >
> > > > > so, i've put the master into the conf/masters file and the
> > conf/slaves
> > > > > file.
> > > > >
> > > > > things seem to work, but there's no DataNode process listed with
> jps
> > > on
> > > > > the
> > > > > master. i'm wondering if there's a switch i need to flip to tell
> > > hadoop
> > > > > to
> > > > > use the master as a datanode even if it's in the slaves file?
> > > > >
> > > > > thanks again.
> > > > >
> > > > > -colin
> > > >
> > > >
> > > >
> >
> >
> >
Re: Master as DataNode
Posted by Colin Freas <co...@gmail.com>.
yup, got it working with that technique.
pushed it out to 5 machines, things look good. appreciate the help.
what is it that causes this? i know i formatted the dfs more than once. is
that what does it? or just adding nodes, or... ?
-colin
On Fri, Mar 21, 2008 at 2:30 PM, Jeff Eastman <je...@windwardsolutions.com>
wrote:
> I encountered this while I was starting out too, while moving from a
> single
> node cluster to more nodes. I suggest clearing your hadoop-datastore
> directory, reformatting the HDFS and restarting again. You are very close
> :)
> Jeff
>
> > -----Original Message-----
> > From: Colin Freas [mailto:colinfreas@gmail.com]
> > Sent: Friday, March 21, 2008 11:18 AM
> > To: core-user@hadoop.apache.org
> > Subject: Re: Master as DataNode
> >
> > ah:
> >
> > 2008-03-21 14:06:05,526 ERROR org.apache.hadoop.dfs.DataNode:
> > java.io.IOException: Incompatible namespaceIDs in
> > /var/tmp/hadoop-datastore/hadoop/dfs/data: namenode namespaceID =
> > 2121666262; datanode namespaceID = 2058961420
> >
> >
> > looks like i'm hitting this "Incompatible namespaceID" bug:
> > http://issues.apache.org/jira/browse/HADOOP-1212
> >
> > is there a work around for this?
> >
> > -colin
> >
> >
> > On Fri, Mar 21, 2008 at 1:50 PM, Jeff Eastman <
> jeff@windwardsolutions.com>
> > wrote:
> >
> > > Check your logs. That should work out of the box with the
> configuration
> > > steps you described.
> > >
> > > Jeff
> > >
> > > > -----Original Message-----
> > > > From: Colin Freas [mailto:colinfreas@gmail.com]
> > > > Sent: Friday, March 21, 2008 10:40 AM
> > > > To: core-user@hadoop.apache.org
> > > > Subject: Master as DataNode
> > > >
> > > > setting up a simple hadoop cluster with two machines, i've gotten to
> > the
> > > > point where the two machines can see each other, things seem fine,
> but
> > > i'm
> > > > trying to set up the master as both a master and a slave, just for
> > > testing
> > > > purposes.
> > > >
> > > > so, i've put the master into the conf/masters file and the
> conf/slaves
> > > > file.
> > > >
> > > > things seem to work, but there's no DataNode process listed with jps
> > on
> > > > the
> > > > master. i'm wondering if there's a switch i need to flip to tell
> > hadoop
> > > > to
> > > > use the master as a datanode even if it's in the slaves file?
> > > >
> > > > thanks again.
> > > >
> > > > -colin
> > >
> > >
> > >
>
>
>
RE: Master as DataNode
Posted by Jeff Eastman <je...@windwardsolutions.com>.
I encountered this while I was starting out too, while moving from a single
node cluster to more nodes. I suggest clearing your hadoop-datastore
directory, reformatting the HDFS and restarting again. You are very close :)
Jeff
> -----Original Message-----
> From: Colin Freas [mailto:colinfreas@gmail.com]
> Sent: Friday, March 21, 2008 11:18 AM
> To: core-user@hadoop.apache.org
> Subject: Re: Master as DataNode
>
> ah:
>
> 2008-03-21 14:06:05,526 ERROR org.apache.hadoop.dfs.DataNode:
> java.io.IOException: Incompatible namespaceIDs in
> /var/tmp/hadoop-datastore/hadoop/dfs/data: namenode namespaceID =
> 2121666262; datanode namespaceID = 2058961420
>
>
> looks like i'm hitting this "Incompatible namespaceID" bug:
> http://issues.apache.org/jira/browse/HADOOP-1212
>
> is there a work around for this?
>
> -colin
>
>
> On Fri, Mar 21, 2008 at 1:50 PM, Jeff Eastman <je...@windwardsolutions.com>
> wrote:
>
> > Check your logs. That should work out of the box with the configuration
> > steps you described.
> >
> > Jeff
> >
> > > -----Original Message-----
> > > From: Colin Freas [mailto:colinfreas@gmail.com]
> > > Sent: Friday, March 21, 2008 10:40 AM
> > > To: core-user@hadoop.apache.org
> > > Subject: Master as DataNode
> > >
> > > setting up a simple hadoop cluster with two machines, i've gotten to
> the
> > > point where the two machines can see each other, things seem fine, but
> > i'm
> > > trying to set up the master as both a master and a slave, just for
> > testing
> > > purposes.
> > >
> > > so, i've put the master into the conf/masters file and the conf/slaves
> > > file.
> > >
> > > things seem to work, but there's no DataNode process listed with jps
> on
> > > the
> > > master. i'm wondering if there's a switch i need to flip to tell
> hadoop
> > > to
> > > use the master as a datanode even if it's in the slaves file?
> > >
> > > thanks again.
> > >
> > > -colin
> >
> >
> >
Re: Master as DataNode
Posted by Colin Freas <co...@gmail.com>.
ah:
2008-03-21 14:06:05,526 ERROR org.apache.hadoop.dfs.DataNode:
java.io.IOException: Incompatible namespaceIDs in
/var/tmp/hadoop-datastore/hadoop/dfs/data: namenode namespaceID =
2121666262; datanode namespaceID = 2058961420
looks like i'm hitting this "Incompatible namespaceID" bug:
http://issues.apache.org/jira/browse/HADOOP-1212
is there a work around for this?
-colin
On Fri, Mar 21, 2008 at 1:50 PM, Jeff Eastman <je...@windwardsolutions.com>
wrote:
> Check your logs. That should work out of the box with the configuration
> steps you described.
>
> Jeff
>
> > -----Original Message-----
> > From: Colin Freas [mailto:colinfreas@gmail.com]
> > Sent: Friday, March 21, 2008 10:40 AM
> > To: core-user@hadoop.apache.org
> > Subject: Master as DataNode
> >
> > setting up a simple hadoop cluster with two machines, i've gotten to the
> > point where the two machines can see each other, things seem fine, but
> i'm
> > trying to set up the master as both a master and a slave, just for
> testing
> > purposes.
> >
> > so, i've put the master into the conf/masters file and the conf/slaves
> > file.
> >
> > things seem to work, but there's no DataNode process listed with jps on
> > the
> > master. i'm wondering if there's a switch i need to flip to tell hadoop
> > to
> > use the master as a datanode even if it's in the slaves file?
> >
> > thanks again.
> >
> > -colin
>
>
>
RE: Master as DataNode
Posted by Jeff Eastman <je...@windwardsolutions.com>.
Check your logs. That should work out of the box with the configuration
steps you described.
Jeff
> -----Original Message-----
> From: Colin Freas [mailto:colinfreas@gmail.com]
> Sent: Friday, March 21, 2008 10:40 AM
> To: core-user@hadoop.apache.org
> Subject: Master as DataNode
>
> setting up a simple hadoop cluster with two machines, i've gotten to the
> point where the two machines can see each other, things seem fine, but i'm
> trying to set up the master as both a master and a slave, just for testing
> purposes.
>
> so, i've put the master into the conf/masters file and the conf/slaves
> file.
>
> things seem to work, but there's no DataNode process listed with jps on
> the
> master. i'm wondering if there's a switch i need to flip to tell hadoop
> to
> use the master as a datanode even if it's in the slaves file?
>
> thanks again.
>
> -colin