You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Fabio A. Miranda" <fa...@gmail.com> on 2010/11/09 19:47:41 UTC
Cluster setup
hello,
I am trying to setup an Hadoop cluster. From the docs, it says I need
two master: NameNode and Jobtracker and one slave: datanode,
tasktracker.
so, I need at least 4 machines to set up a cluster with hadoop ?
How can I define the role of each machine if core-site.xml needs to be
the same ?
regards,
fabio.
Re: Cluster setup
Posted by Steve Loughran <st...@apache.org>.
On 09/11/10 18:47, Fabio A. Miranda wrote:
> hello,
>
> I am trying to setup an Hadoop cluster. From the docs, it says I need
> two master: NameNode and Jobtracker and one slave: datanode,
> tasktracker.
>
> so, I need at least 4 machines to set up a cluster with hadoop ?
>
> How can I define the role of each machine if core-site.xml needs to be
> the same ?
You can put the JT on the same server as the namenode; this is something
doing in any "small" cluster, where small probably means less than 20 or
so servers.
I do virtual clusters, and make sure that the "master" node has a very
small datanode on it too (but no job tracker), as with that setup the
filesystem and the Job tracker can go live on a single VM -you only add
more virtual worker nodes to get more filesystem capacity and task trackers.
-steve
Re: Cluster setup
Posted by James Seigel <ja...@tynt.com>.
You can start with one machine or one VM if you are just looking to try out hadoop.
James.
On 2010-11-09, at 11:47 AM, Fabio A. Miranda wrote:
> hello,
>
> I am trying to setup an Hadoop cluster. From the docs, it says I need
> two master: NameNode and Jobtracker and one slave: datanode,
> tasktracker.
>
> so, I need at least 4 machines to set up a cluster with hadoop ?
>
> How can I define the role of each machine if core-site.xml needs to be
> the same ?
>
> regards,
>
> fabio.
>
>
Re: Cluster setup
Posted by Aaron Eng <ae...@maprtech.com>.
Hi Fabio,
I found this site extremely helpful in explaining how to do a one node setup
for a first time user:
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29
On Tue, Nov 9, 2010 at 10:54 AM, Fabio A. Miranda <fabio.a.miranda@gmail.com
> wrote:
> Hello,
>
>
> > You don't need 4 machines. You can put everything on a single machine.
> That is the easiest to get started. Once you have a cluster running on a
> single machine then you can spread out over multiple machines.
> >
>
> Ok, but I don't understand what files exactly do I need to set up ?
>
> Can I get sample .xml files for basic cluster setup ?
>
>
>
>
RE: Cluster setup
Posted by "Habermaas, William" <Wi...@fatwire.com>.
Visit the quickstart page and setup pseudo distributed mode on a single machine.
http://hadoop.apache.org/common/docs/r0.20.0/quickstart.html
Bill
-----Original Message-----
From: Fabio A. Miranda [mailto:fabio.a.miranda@gmail.com]
Sent: Tuesday, November 09, 2010 1:54 PM
To: common-user@hadoop.apache.org
Subject: RE: Cluster setup
Hello,
> You don't need 4 machines. You can put everything on a single machine. That is the easiest to get started. Once you have a cluster running on a single machine then you can spread out over multiple machines.
>
Ok, but I don't understand what files exactly do I need to set up ?
Can I get sample .xml files for basic cluster setup ?
RE: Cluster setup
Posted by "Fabio A. Miranda" <fa...@gmail.com>.
Hello,
> You don't need 4 machines. You can put everything on a single machine. That is the easiest to get started. Once you have a cluster running on a single machine then you can spread out over multiple machines.
>
Ok, but I don't understand what files exactly do I need to set up ?
Can I get sample .xml files for basic cluster setup ?
RE: Cluster setup
Posted by "Habermaas, William" <Wi...@fatwire.com>.
Fabio,
You don't need 4 machines. You can put everything on a single machine. That is the easiest to get started. Once you have a cluster running on a single machine then you can spread out over multiple machines.
Best,
Bill
-----Original Message-----
From: Fabio A. Miranda [mailto:fabio.a.miranda@gmail.com]
Sent: Tuesday, November 09, 2010 1:48 PM
To: common-user@hadoop.apache.org
Subject: Cluster setup
hello,
I am trying to setup an Hadoop cluster. From the docs, it says I need
two master: NameNode and Jobtracker and one slave: datanode,
tasktracker.
so, I need at least 4 machines to set up a cluster with hadoop ?
How can I define the role of each machine if core-site.xml needs to be
the same ?
regards,
fabio.