You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Michael Namaiandeh <mn...@healthcit.com> on 2012/11/27 21:15:32 UTC
Hadoop cluster configuration
Hi Hadoop user community,
I am trying to setup my first Hadoop cluster and I've found most of the instructions a little confusing. I've seen how-to's that say "core-site.xml" should have hdfs://localhost:8020 and others say hdfs://localhost:50030". Which one is correct? Can someone please help with the configuration?
I'm starting out with a 4 machine cluster on CentOS VM's. I've created the 2 masters and 2 slaves.
My Masters are:
JobTracker
NameNode
My Slaves are:
TrackTracker
DataNode
Finally, how can I test to verify that the masters are talking to the slaves and vice-versa? Thanks in advance.
Re: Hadoop cluster configuration
Posted by Visioner Sadak <vi...@gmail.com>.
could you tell me which version of hadoop r u using...
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
Re: Hadoop cluster configuration
Posted by Visioner Sadak <vi...@gmail.com>.
could you tell me which version of hadoop r u using...
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
Re: Hadoop cluster configuration
Posted by Dino Kečo <di...@gmail.com>.
Hi Michael,
There is good guide how to setup multinode hadoop cluster.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
Hope it helps.
Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851
On Tue, Nov 27, 2012 at 9:21 PM, Mohammad Tariq <do...@gmail.com> wrote:
> a pa
Re: Hadoop cluster configuration
Posted by Dino Kečo <di...@gmail.com>.
Hi Michael,
There is good guide how to setup multinode hadoop cluster.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
Hope it helps.
Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851
On Tue, Nov 27, 2012 at 9:21 PM, Mohammad Tariq <do...@gmail.com> wrote:
> a pa
Re: Hadoop cluster configuration
Posted by Dino Kečo <di...@gmail.com>.
Hi Michael,
There is good guide how to setup multinode hadoop cluster.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
Hope it helps.
Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851
On Tue, Nov 27, 2012 at 9:21 PM, Mohammad Tariq <do...@gmail.com> wrote:
> a pa
Re: Hadoop cluster configuration
Posted by Dino Kečo <di...@gmail.com>.
Hi Michael,
There is good guide how to setup multinode hadoop cluster.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
Hope it helps.
Regards,
Dino Kečo
msn: xdinno@hotmail.com
mail: dino.keco@gmail.com
skype: dino.keco
phone: +387 61 507 851
On Tue, Nov 27, 2012 at 9:21 PM, Mohammad Tariq <do...@gmail.com> wrote:
> a pa
Re: Hadoop cluster configuration
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Michael,
You can use any port of your choice. But I would suggest not to use 50030
or 50070 as they are the default ports for Hdfs and MR webUI. Also, if you
are planning to create a distributed cluster (4 nodes as specified by you),
do not use localhost anywhere. Instead use appropriate hostnames. Make sure
you do not have any DNS resolution issue. It could really become a pain.
Regards,
Mohammad Tariq
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
Re: Hadoop cluster configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi Michael,
I'd suggest following Michael Noll's write-up on this topic, at
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/.
It should clear some of your confusion, and also get you up and running
quickly.
P.s. Though the title mentions "Ubuntu", there's very little of Ubuntu
specific information in it, and you should be able to follow it for CentOS
as well.
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
--
Harsh J
Re: Hadoop cluster configuration
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Michael,
You can use any port of your choice. But I would suggest not to use 50030
or 50070 as they are the default ports for Hdfs and MR webUI. Also, if you
are planning to create a distributed cluster (4 nodes as specified by you),
do not use localhost anywhere. Instead use appropriate hostnames. Make sure
you do not have any DNS resolution issue. It could really become a pain.
Regards,
Mohammad Tariq
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
Re: Hadoop cluster configuration
Posted by Visioner Sadak <vi...@gmail.com>.
could you tell me which version of hadoop r u using...
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
Re: Hadoop cluster configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi Michael,
I'd suggest following Michael Noll's write-up on this topic, at
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/.
It should clear some of your confusion, and also get you up and running
quickly.
P.s. Though the title mentions "Ubuntu", there's very little of Ubuntu
specific information in it, and you should be able to follow it for CentOS
as well.
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
--
Harsh J
Re: Hadoop cluster configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi Michael,
I'd suggest following Michael Noll's write-up on this topic, at
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/.
It should clear some of your confusion, and also get you up and running
quickly.
P.s. Though the title mentions "Ubuntu", there's very little of Ubuntu
specific information in it, and you should be able to follow it for CentOS
as well.
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
--
Harsh J
Re: Hadoop cluster configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi Michael,
I'd suggest following Michael Noll's write-up on this topic, at
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/.
It should clear some of your confusion, and also get you up and running
quickly.
P.s. Though the title mentions "Ubuntu", there's very little of Ubuntu
specific information in it, and you should be able to follow it for CentOS
as well.
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
--
Harsh J
Re: Hadoop cluster configuration
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Michael,
You can use any port of your choice. But I would suggest not to use 50030
or 50070 as they are the default ports for Hdfs and MR webUI. Also, if you
are planning to create a distributed cluster (4 nodes as specified by you),
do not use localhost anywhere. Instead use appropriate hostnames. Make sure
you do not have any DNS resolution issue. It could really become a pain.
Regards,
Mohammad Tariq
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
Re: Hadoop cluster configuration
Posted by Visioner Sadak <vi...@gmail.com>.
could you tell me which version of hadoop r u using...
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>
Re: Hadoop cluster configuration
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Michael,
You can use any port of your choice. But I would suggest not to use 50030
or 50070 as they are the default ports for Hdfs and MR webUI. Also, if you
are planning to create a distributed cluster (4 nodes as specified by you),
do not use localhost anywhere. Instead use appropriate hostnames. Make sure
you do not have any DNS resolution issue. It could really become a pain.
Regards,
Mohammad Tariq
On Wed, Nov 28, 2012 at 1:45 AM, Michael Namaiandeh <
mnamaiandeh@healthcit.com> wrote:
> Hi Hadoop user community,****
>
> ** **
>
> I am trying to setup my first Hadoop cluster and I’ve found most of the
> instructions a little confusing. I’ve seen how-to’s that say
> “core-site.xml” should have hdfs://localhost:8020 and others say
> hdfs://localhost:50030”. Which one is correct? Can someone please help with
> the configuration? ****
>
> ** **
>
> I’m starting out with a 4 machine cluster on CentOS VM’s. I’ve created the
> 2 masters and 2 slaves.****
>
> ** **
>
> My Masters are:****
>
> JobTracker****
>
> NameNode****
>
> ** **
>
> My Slaves are:****
>
> TrackTracker****
>
> DataNode****
>
> ** **
>
> Finally, how can I test to verify that the masters are talking to the
> slaves and vice-versa? Thanks in advance.****
>