You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Humayun kabir <hu...@gmail.com> on 2011/12/22 10:24:48 UTC
Hadoop configuration
someone please help me to configure hadoop such as core-site.xml,
hdfs-site.xml, mapred-site.xml etc.
please provide some example. it is badly needed. because i run in a 2 node
cluster. when i run the wordcount example then it gives the result too
mutch fetch failure.
Re: Hadoop configuration
Posted by alo alt <wg...@googlemail.com>.
Hi,
Apache:
http://hadoop.apache.org/common/docs/current/cluster_setup.html
RHEL / CentOS:
http://mapredit.blogspot.com/p/get-hadoop-cluster-running-in-20.html
Ubuntu:
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
- Alex
On Thu, Dec 22, 2011 at 10:24 AM, Humayun kabir <hu...@gmail.com> wrote:
> someone please help me to configure hadoop such as core-site.xml,
> hdfs-site.xml, mapred-site.xml etc.
> please provide some example. it is badly needed. because i run in a 2 node
> cluster. when i run the wordcount example then it gives the result too
> mutch fetch failure.
--
Alexander Lorenz
http://mapredit.blogspot.com
P Think of the environment: please don't print this email unless you
really need to.
Re: Hadoop configuration
Posted by raghavendhra rahul <ra...@gmail.com>.
http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
This is the easiest guide to configure hadoop.
On Thu, Dec 22, 2011 at 2:54 PM, Humayun kabir <hu...@gmail.com>wrote:
> someone please help me to configure hadoop such as core-site.xml,
> hdfs-site.xml, mapred-site.xml etc.
> please provide some example. it is badly needed. because i run in a 2 node
> cluster. when i run the wordcount example then it gives the result too
> mutch fetch failure.
>
Re: Hadoop configuration
Posted by Humayun kabir <hu...@gmail.com>.
Hi Uma,
Thanks a lot. At last it is running without errors. Thank you very much for
your suggestion.
On 26 December 2011 20:04, Uma Maheswara Rao G <ma...@huawei.com> wrote:
> Hey Humayun,
> Looks your hostname still not resoling properly. even though you
> configured hostnames as master, slave...etc, it is getting humayun as
> hostname.
> just edit /etc/HOSTNAME file with correct hostname what you are expecting
> here.
> To confirm whether it is resolving properly or not, you can just do below
> steps
> #hostname
> ............................//should get hostname here
> correctly ( ex: master)
> #hostname -i
> ..........................//should resolve correct IP
> here ... ( ex: master ip)
>
>
> and make sure slave and slave1 sre pingable each other.
>
> Regards,
> Uma
>
> ________________________________________
> From: Humayun kabir [humayun0156@gmail.com]
> Sent: Saturday, December 24, 2011 9:51 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Hadoop configuration
>
> i've checked my log files. But i don't understand to why this error occurs.
> here i my logs files. please give me some suggestion.
>
> jobtracker.log < http://paste.ubuntu.com/781181/ >
>
> namenode.log < http://paste.ubuntu.com/781183/ >
>
> datanode.log(1st machine) < http://paste.ubuntu.com/781176/ >
>
> datanode.log(2nd machine) < http://paste.ubuntu.com/781195
> <goog_2054845717>/
> >
>
> tasktracker.log(1st machine) < http://paste.ubuntu.com/781192/ >
>
> tasktracker.log(2nd machine) < http://paste.ubuntu.com/781197/ >
>
>
>
> On 24 December 2011 15:26, Joey Krabacher <jk...@gmail.com> wrote:
>
> > have you checked your log files for any clues?
> >
> > --Joey
> >
> > On Sat, Dec 24, 2011 at 3:15 AM, Humayun kabir <hu...@gmail.com>
> > wrote:
> > > Hi Uma,
> > >
> > > Thank you very much for your tips. We tried it in 3 nodes in virtual
> box
> > as
> > > you suggested. But still we are facing problem. Here is our all
> > > configuration file to all nodes. please take a look and show us some
> ways
> > > to solve it. It was nice and it would be great if you help us in this
> > > regards.
> > >
> > > <http://core-site.xml>core-site.xml < http://pastebin.com/Twn5edrp >
> > > hdfs-site.xml < http://pastebin.com/k4hR4GE9 >
> > > mapred-site.xml < http://pastebin.com/gZuyHswS >
> > >
> > > /etc/hosts < http://pastebin.com/5s0yhgnj >
> > >
> > > output < http://paste.ubuntu.com/780807/ >
> > >
> > >
> > > Hope you will understand and extend your helping hand towards us.
> > >
> > > Have a nice day.
> > >
> > > Regards
> > > Humayun
> > >
> > > On 23 December 2011 17:31, Uma Maheswara Rao G <ma...@huawei.com>
> > wrote:
> > >
> > >> Hi Humayun ,
> > >>
> > >> Lets assume you have JT, TT1, TT2, TT3
> > >>
> > >> Now you should configure the \etc\hosts like below examle
> > >>
> > >> 10.18.xx.1 JT
> > >>
> > >> 10.18.xx.2 TT1
> > >>
> > >> 10.18.xx.3 TT2
> > >>
> > >> 10.18.xx.4 TT3
> > >>
> > >> Configure the same set in all the machines, so that all task
> trackers
> > >> can talk each other with hostnames correctly. Also pls remove some
> > entries
> > >> from your files
> > >>
> > >> 127.0.0.1 localhost.localdomain localhost
> > >>
> > >> 127.0.1.1 humayun
> > >>
> > >>
> > >>
> > >> I have seen others already suggested many links for the regular
> > >> configuration items. Hope you might clear about them.
> > >>
> > >> hope it will help...
> > >>
> > >> Regards,
> > >>
> > >> Uma
> > >>
> > >> ________________________________
> > >>
> > >> From: Humayun kabir [humayun0156@gmail.com]
> > >> Sent: Thursday, December 22, 2011 10:34 PM
> > >> To: common-user@hadoop.apache.org; Uma Maheswara Rao G
> > >> Subject: Re: Hadoop configuration
> > >>
> > >> Hello Uma,
> > >>
> > >> Thanks for your cordial and quick reply. It would be great if you
> > explain
> > >> what you suggested to do. Right now we are running on following
> > >> configuration.
> > >>
> > >> We are using hadoop on virtual box. when it is a single node then it
> > works
> > >> fine for big dataset larger than the default block size. but in case
> of
> > >> multinode cluster (2 nodes) we are facing some problems. We are able
> to
> > >> ping both "Master->Slave" and "Slave->Master".
> > >> Like when the input dataset is smaller than the default block size(64
> > MB)
> > >> then it works fine. but when the input dataset is larger than the
> > default
> > >> block size then it shows ‘too much fetch failure’ in reduce state.
> > >> here is the output link
> > >> http://paste.ubuntu.com/707517/
> > >>
> > >> this is our /etc/hosts file
> > >>
> > >> 192.168.60.147 humayun # Added by NetworkManager
> > >> 127.0.0.1 localhost.localdomain localhost
> > >> ::1 humayun localhost6.localdomain6 localhost6
> > >> 127.0.1.1 humayun
> > >>
> > >> # The following lines are desirable for IPv6 capable hosts
> > >> ::1 localhost ip6-localhost ip6-loopback
> > >> fe00::0 ip6-localnet
> > >> ff00::0 ip6-mcastprefix
> > >> ff02::1 ip6-allnodes
> > >> ff02::2 ip6-allrouters
> > >> ff02::3 ip6-allhosts
> > >>
> > >> 192.168.60.1 master
> > >> 192.168.60.2 slave
> > >>
> > >>
> > >> Regards,
> > >>
> > >> -Humayun.
> > >>
> > >>
> > >> On 22 December 2011 15:47, Uma Maheswara Rao G <maheswara@huawei.com
> > >> <ma...@huawei.com>> wrote:
> > >> Hey Humayun,
> > >>
> > >> To solve the too many fetch failures problem, you should configure
> host
> > >> mapping correctly.
> > >> Each tasktracker should be able to ping from each other.
> > >>
> > >> Regards,
> > >> Uma
> > >> ________________________________________
> > >> From: Humayun kabir [humayun0156@gmail.com<mailto:
> humayun0156@gmail.com
> > >]
> > >> Sent: Thursday, December 22, 2011 2:54 PM
> > >> To: common-user@hadoop.apache.org<mailto:
> common-user@hadoop.apache.org>
> > >> Subject: Hadoop configuration
> > >>
> > >> someone please help me to configure hadoop such as core-site.xml,
> > >> hdfs-site.xml, mapred-site.xml etc.
> > >> please provide some example. it is badly needed. because i run in a 2
> > node
> > >> cluster. when i run the wordcount example then it gives the result too
> > >> mutch fetch failure.
> > >>
> > >>
> >
>
RE: Hadoop configuration
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
Hey Humayun,
Looks your hostname still not resoling properly. even though you configured hostnames as master, slave...etc, it is getting humayun as hostname.
just edit /etc/HOSTNAME file with correct hostname what you are expecting here.
To confirm whether it is resolving properly or not, you can just do below steps
#hostname
............................//should get hostname here correctly ( ex: master)
#hostname -i
..........................//should resolve correct IP here ... ( ex: master ip)
and make sure slave and slave1 sre pingable each other.
Regards,
Uma
________________________________________
From: Humayun kabir [humayun0156@gmail.com]
Sent: Saturday, December 24, 2011 9:51 PM
To: common-user@hadoop.apache.org
Subject: Re: Hadoop configuration
i've checked my log files. But i don't understand to why this error occurs.
here i my logs files. please give me some suggestion.
jobtracker.log < http://paste.ubuntu.com/781181/ >
namenode.log < http://paste.ubuntu.com/781183/ >
datanode.log(1st machine) < http://paste.ubuntu.com/781176/ >
datanode.log(2nd machine) < http://paste.ubuntu.com/781195<goog_2054845717>/
>
tasktracker.log(1st machine) < http://paste.ubuntu.com/781192/ >
tasktracker.log(2nd machine) < http://paste.ubuntu.com/781197/ >
On 24 December 2011 15:26, Joey Krabacher <jk...@gmail.com> wrote:
> have you checked your log files for any clues?
>
> --Joey
>
> On Sat, Dec 24, 2011 at 3:15 AM, Humayun kabir <hu...@gmail.com>
> wrote:
> > Hi Uma,
> >
> > Thank you very much for your tips. We tried it in 3 nodes in virtual box
> as
> > you suggested. But still we are facing problem. Here is our all
> > configuration file to all nodes. please take a look and show us some ways
> > to solve it. It was nice and it would be great if you help us in this
> > regards.
> >
> > <http://core-site.xml>core-site.xml < http://pastebin.com/Twn5edrp >
> > hdfs-site.xml < http://pastebin.com/k4hR4GE9 >
> > mapred-site.xml < http://pastebin.com/gZuyHswS >
> >
> > /etc/hosts < http://pastebin.com/5s0yhgnj >
> >
> > output < http://paste.ubuntu.com/780807/ >
> >
> >
> > Hope you will understand and extend your helping hand towards us.
> >
> > Have a nice day.
> >
> > Regards
> > Humayun
> >
> > On 23 December 2011 17:31, Uma Maheswara Rao G <ma...@huawei.com>
> wrote:
> >
> >> Hi Humayun ,
> >>
> >> Lets assume you have JT, TT1, TT2, TT3
> >>
> >> Now you should configure the \etc\hosts like below examle
> >>
> >> 10.18.xx.1 JT
> >>
> >> 10.18.xx.2 TT1
> >>
> >> 10.18.xx.3 TT2
> >>
> >> 10.18.xx.4 TT3
> >>
> >> Configure the same set in all the machines, so that all task trackers
> >> can talk each other with hostnames correctly. Also pls remove some
> entries
> >> from your files
> >>
> >> 127.0.0.1 localhost.localdomain localhost
> >>
> >> 127.0.1.1 humayun
> >>
> >>
> >>
> >> I have seen others already suggested many links for the regular
> >> configuration items. Hope you might clear about them.
> >>
> >> hope it will help...
> >>
> >> Regards,
> >>
> >> Uma
> >>
> >> ________________________________
> >>
> >> From: Humayun kabir [humayun0156@gmail.com]
> >> Sent: Thursday, December 22, 2011 10:34 PM
> >> To: common-user@hadoop.apache.org; Uma Maheswara Rao G
> >> Subject: Re: Hadoop configuration
> >>
> >> Hello Uma,
> >>
> >> Thanks for your cordial and quick reply. It would be great if you
> explain
> >> what you suggested to do. Right now we are running on following
> >> configuration.
> >>
> >> We are using hadoop on virtual box. when it is a single node then it
> works
> >> fine for big dataset larger than the default block size. but in case of
> >> multinode cluster (2 nodes) we are facing some problems. We are able to
> >> ping both "Master->Slave" and "Slave->Master".
> >> Like when the input dataset is smaller than the default block size(64
> MB)
> >> then it works fine. but when the input dataset is larger than the
> default
> >> block size then it shows ‘too much fetch failure’ in reduce state.
> >> here is the output link
> >> http://paste.ubuntu.com/707517/
> >>
> >> this is our /etc/hosts file
> >>
> >> 192.168.60.147 humayun # Added by NetworkManager
> >> 127.0.0.1 localhost.localdomain localhost
> >> ::1 humayun localhost6.localdomain6 localhost6
> >> 127.0.1.1 humayun
> >>
> >> # The following lines are desirable for IPv6 capable hosts
> >> ::1 localhost ip6-localhost ip6-loopback
> >> fe00::0 ip6-localnet
> >> ff00::0 ip6-mcastprefix
> >> ff02::1 ip6-allnodes
> >> ff02::2 ip6-allrouters
> >> ff02::3 ip6-allhosts
> >>
> >> 192.168.60.1 master
> >> 192.168.60.2 slave
> >>
> >>
> >> Regards,
> >>
> >> -Humayun.
> >>
> >>
> >> On 22 December 2011 15:47, Uma Maheswara Rao G <maheswara@huawei.com
> >> <ma...@huawei.com>> wrote:
> >> Hey Humayun,
> >>
> >> To solve the too many fetch failures problem, you should configure host
> >> mapping correctly.
> >> Each tasktracker should be able to ping from each other.
> >>
> >> Regards,
> >> Uma
> >> ________________________________________
> >> From: Humayun kabir [humayun0156@gmail.com<mailto:humayun0156@gmail.com
> >]
> >> Sent: Thursday, December 22, 2011 2:54 PM
> >> To: common-user@hadoop.apache.org<ma...@hadoop.apache.org>
> >> Subject: Hadoop configuration
> >>
> >> someone please help me to configure hadoop such as core-site.xml,
> >> hdfs-site.xml, mapred-site.xml etc.
> >> please provide some example. it is badly needed. because i run in a 2
> node
> >> cluster. when i run the wordcount example then it gives the result too
> >> mutch fetch failure.
> >>
> >>
>
Re: Hadoop configuration
Posted by Humayun kabir <hu...@gmail.com>.
i've checked my log files. But i don't understand to why this error occurs.
here i my logs files. please give me some suggestion.
jobtracker.log < http://paste.ubuntu.com/781181/ >
namenode.log < http://paste.ubuntu.com/781183/ >
datanode.log(1st machine) < http://paste.ubuntu.com/781176/ >
datanode.log(2nd machine) < http://paste.ubuntu.com/781195<goog_2054845717>/
>
tasktracker.log(1st machine) < http://paste.ubuntu.com/781192/ >
tasktracker.log(2nd machine) < http://paste.ubuntu.com/781197/ >
On 24 December 2011 15:26, Joey Krabacher <jk...@gmail.com> wrote:
> have you checked your log files for any clues?
>
> --Joey
>
> On Sat, Dec 24, 2011 at 3:15 AM, Humayun kabir <hu...@gmail.com>
> wrote:
> > Hi Uma,
> >
> > Thank you very much for your tips. We tried it in 3 nodes in virtual box
> as
> > you suggested. But still we are facing problem. Here is our all
> > configuration file to all nodes. please take a look and show us some ways
> > to solve it. It was nice and it would be great if you help us in this
> > regards.
> >
> > <http://core-site.xml>core-site.xml < http://pastebin.com/Twn5edrp >
> > hdfs-site.xml < http://pastebin.com/k4hR4GE9 >
> > mapred-site.xml < http://pastebin.com/gZuyHswS >
> >
> > /etc/hosts < http://pastebin.com/5s0yhgnj >
> >
> > output < http://paste.ubuntu.com/780807/ >
> >
> >
> > Hope you will understand and extend your helping hand towards us.
> >
> > Have a nice day.
> >
> > Regards
> > Humayun
> >
> > On 23 December 2011 17:31, Uma Maheswara Rao G <ma...@huawei.com>
> wrote:
> >
> >> Hi Humayun ,
> >>
> >> Lets assume you have JT, TT1, TT2, TT3
> >>
> >> Now you should configure the \etc\hosts like below examle
> >>
> >> 10.18.xx.1 JT
> >>
> >> 10.18.xx.2 TT1
> >>
> >> 10.18.xx.3 TT2
> >>
> >> 10.18.xx.4 TT3
> >>
> >> Configure the same set in all the machines, so that all task trackers
> >> can talk each other with hostnames correctly. Also pls remove some
> entries
> >> from your files
> >>
> >> 127.0.0.1 localhost.localdomain localhost
> >>
> >> 127.0.1.1 humayun
> >>
> >>
> >>
> >> I have seen others already suggested many links for the regular
> >> configuration items. Hope you might clear about them.
> >>
> >> hope it will help...
> >>
> >> Regards,
> >>
> >> Uma
> >>
> >> ________________________________
> >>
> >> From: Humayun kabir [humayun0156@gmail.com]
> >> Sent: Thursday, December 22, 2011 10:34 PM
> >> To: common-user@hadoop.apache.org; Uma Maheswara Rao G
> >> Subject: Re: Hadoop configuration
> >>
> >> Hello Uma,
> >>
> >> Thanks for your cordial and quick reply. It would be great if you
> explain
> >> what you suggested to do. Right now we are running on following
> >> configuration.
> >>
> >> We are using hadoop on virtual box. when it is a single node then it
> works
> >> fine for big dataset larger than the default block size. but in case of
> >> multinode cluster (2 nodes) we are facing some problems. We are able to
> >> ping both "Master->Slave" and "Slave->Master".
> >> Like when the input dataset is smaller than the default block size(64
> MB)
> >> then it works fine. but when the input dataset is larger than the
> default
> >> block size then it shows ‘too much fetch failure’ in reduce state.
> >> here is the output link
> >> http://paste.ubuntu.com/707517/
> >>
> >> this is our /etc/hosts file
> >>
> >> 192.168.60.147 humayun # Added by NetworkManager
> >> 127.0.0.1 localhost.localdomain localhost
> >> ::1 humayun localhost6.localdomain6 localhost6
> >> 127.0.1.1 humayun
> >>
> >> # The following lines are desirable for IPv6 capable hosts
> >> ::1 localhost ip6-localhost ip6-loopback
> >> fe00::0 ip6-localnet
> >> ff00::0 ip6-mcastprefix
> >> ff02::1 ip6-allnodes
> >> ff02::2 ip6-allrouters
> >> ff02::3 ip6-allhosts
> >>
> >> 192.168.60.1 master
> >> 192.168.60.2 slave
> >>
> >>
> >> Regards,
> >>
> >> -Humayun.
> >>
> >>
> >> On 22 December 2011 15:47, Uma Maheswara Rao G <maheswara@huawei.com
> >> <ma...@huawei.com>> wrote:
> >> Hey Humayun,
> >>
> >> To solve the too many fetch failures problem, you should configure host
> >> mapping correctly.
> >> Each tasktracker should be able to ping from each other.
> >>
> >> Regards,
> >> Uma
> >> ________________________________________
> >> From: Humayun kabir [humayun0156@gmail.com<mailto:humayun0156@gmail.com
> >]
> >> Sent: Thursday, December 22, 2011 2:54 PM
> >> To: common-user@hadoop.apache.org<ma...@hadoop.apache.org>
> >> Subject: Hadoop configuration
> >>
> >> someone please help me to configure hadoop such as core-site.xml,
> >> hdfs-site.xml, mapred-site.xml etc.
> >> please provide some example. it is badly needed. because i run in a 2
> node
> >> cluster. when i run the wordcount example then it gives the result too
> >> mutch fetch failure.
> >>
> >>
>
Re: Hadoop configuration
Posted by Joey Krabacher <jk...@gmail.com>.
have you checked your log files for any clues?
--Joey
On Sat, Dec 24, 2011 at 3:15 AM, Humayun kabir <hu...@gmail.com> wrote:
> Hi Uma,
>
> Thank you very much for your tips. We tried it in 3 nodes in virtual box as
> you suggested. But still we are facing problem. Here is our all
> configuration file to all nodes. please take a look and show us some ways
> to solve it. It was nice and it would be great if you help us in this
> regards.
>
> <http://core-site.xml>core-site.xml < http://pastebin.com/Twn5edrp >
> hdfs-site.xml < http://pastebin.com/k4hR4GE9 >
> mapred-site.xml < http://pastebin.com/gZuyHswS >
>
> /etc/hosts < http://pastebin.com/5s0yhgnj >
>
> output < http://paste.ubuntu.com/780807/ >
>
>
> Hope you will understand and extend your helping hand towards us.
>
> Have a nice day.
>
> Regards
> Humayun
>
> On 23 December 2011 17:31, Uma Maheswara Rao G <ma...@huawei.com> wrote:
>
>> Hi Humayun ,
>>
>> Lets assume you have JT, TT1, TT2, TT3
>>
>> Now you should configure the \etc\hosts like below examle
>>
>> 10.18.xx.1 JT
>>
>> 10.18.xx.2 TT1
>>
>> 10.18.xx.3 TT2
>>
>> 10.18.xx.4 TT3
>>
>> Configure the same set in all the machines, so that all task trackers
>> can talk each other with hostnames correctly. Also pls remove some entries
>> from your files
>>
>> 127.0.0.1 localhost.localdomain localhost
>>
>> 127.0.1.1 humayun
>>
>>
>>
>> I have seen others already suggested many links for the regular
>> configuration items. Hope you might clear about them.
>>
>> hope it will help...
>>
>> Regards,
>>
>> Uma
>>
>> ________________________________
>>
>> From: Humayun kabir [humayun0156@gmail.com]
>> Sent: Thursday, December 22, 2011 10:34 PM
>> To: common-user@hadoop.apache.org; Uma Maheswara Rao G
>> Subject: Re: Hadoop configuration
>>
>> Hello Uma,
>>
>> Thanks for your cordial and quick reply. It would be great if you explain
>> what you suggested to do. Right now we are running on following
>> configuration.
>>
>> We are using hadoop on virtual box. when it is a single node then it works
>> fine for big dataset larger than the default block size. but in case of
>> multinode cluster (2 nodes) we are facing some problems. We are able to
>> ping both "Master->Slave" and "Slave->Master".
>> Like when the input dataset is smaller than the default block size(64 MB)
>> then it works fine. but when the input dataset is larger than the default
>> block size then it shows ‘too much fetch failure’ in reduce state.
>> here is the output link
>> http://paste.ubuntu.com/707517/
>>
>> this is our /etc/hosts file
>>
>> 192.168.60.147 humayun # Added by NetworkManager
>> 127.0.0.1 localhost.localdomain localhost
>> ::1 humayun localhost6.localdomain6 localhost6
>> 127.0.1.1 humayun
>>
>> # The following lines are desirable for IPv6 capable hosts
>> ::1 localhost ip6-localhost ip6-loopback
>> fe00::0 ip6-localnet
>> ff00::0 ip6-mcastprefix
>> ff02::1 ip6-allnodes
>> ff02::2 ip6-allrouters
>> ff02::3 ip6-allhosts
>>
>> 192.168.60.1 master
>> 192.168.60.2 slave
>>
>>
>> Regards,
>>
>> -Humayun.
>>
>>
>> On 22 December 2011 15:47, Uma Maheswara Rao G <maheswara@huawei.com
>> <ma...@huawei.com>> wrote:
>> Hey Humayun,
>>
>> To solve the too many fetch failures problem, you should configure host
>> mapping correctly.
>> Each tasktracker should be able to ping from each other.
>>
>> Regards,
>> Uma
>> ________________________________________
>> From: Humayun kabir [humayun0156@gmail.com<ma...@gmail.com>]
>> Sent: Thursday, December 22, 2011 2:54 PM
>> To: common-user@hadoop.apache.org<ma...@hadoop.apache.org>
>> Subject: Hadoop configuration
>>
>> someone please help me to configure hadoop such as core-site.xml,
>> hdfs-site.xml, mapred-site.xml etc.
>> please provide some example. it is badly needed. because i run in a 2 node
>> cluster. when i run the wordcount example then it gives the result too
>> mutch fetch failure.
>>
>>
Re: Hadoop configuration
Posted by Humayun kabir <hu...@gmail.com>.
Hi Uma,
Thank you very much for your tips. We tried it in 3 nodes in virtual box as
you suggested. But still we are facing problem. Here is our all
configuration file to all nodes. please take a look and show us some ways
to solve it. It was nice and it would be great if you help us in this
regards.
<http://core-site.xml>core-site.xml < http://pastebin.com/Twn5edrp >
hdfs-site.xml < http://pastebin.com/k4hR4GE9 >
mapred-site.xml < http://pastebin.com/gZuyHswS >
/etc/hosts < http://pastebin.com/5s0yhgnj >
output < http://paste.ubuntu.com/780807/ >
Hope you will understand and extend your helping hand towards us.
Have a nice day.
Regards
Humayun
On 23 December 2011 17:31, Uma Maheswara Rao G <ma...@huawei.com> wrote:
> Hi Humayun ,
>
> Lets assume you have JT, TT1, TT2, TT3
>
> Now you should configure the \etc\hosts like below examle
>
> 10.18.xx.1 JT
>
> 10.18.xx.2 TT1
>
> 10.18.xx.3 TT2
>
> 10.18.xx.4 TT3
>
> Configure the same set in all the machines, so that all task trackers
> can talk each other with hostnames correctly. Also pls remove some entries
> from your files
>
> 127.0.0.1 localhost.localdomain localhost
>
> 127.0.1.1 humayun
>
>
>
> I have seen others already suggested many links for the regular
> configuration items. Hope you might clear about them.
>
> hope it will help...
>
> Regards,
>
> Uma
>
> ________________________________
>
> From: Humayun kabir [humayun0156@gmail.com]
> Sent: Thursday, December 22, 2011 10:34 PM
> To: common-user@hadoop.apache.org; Uma Maheswara Rao G
> Subject: Re: Hadoop configuration
>
> Hello Uma,
>
> Thanks for your cordial and quick reply. It would be great if you explain
> what you suggested to do. Right now we are running on following
> configuration.
>
> We are using hadoop on virtual box. when it is a single node then it works
> fine for big dataset larger than the default block size. but in case of
> multinode cluster (2 nodes) we are facing some problems. We are able to
> ping both "Master->Slave" and "Slave->Master".
> Like when the input dataset is smaller than the default block size(64 MB)
> then it works fine. but when the input dataset is larger than the default
> block size then it shows ‘too much fetch failure’ in reduce state.
> here is the output link
> http://paste.ubuntu.com/707517/
>
> this is our /etc/hosts file
>
> 192.168.60.147 humayun # Added by NetworkManager
> 127.0.0.1 localhost.localdomain localhost
> ::1 humayun localhost6.localdomain6 localhost6
> 127.0.1.1 humayun
>
> # The following lines are desirable for IPv6 capable hosts
> ::1 localhost ip6-localhost ip6-loopback
> fe00::0 ip6-localnet
> ff00::0 ip6-mcastprefix
> ff02::1 ip6-allnodes
> ff02::2 ip6-allrouters
> ff02::3 ip6-allhosts
>
> 192.168.60.1 master
> 192.168.60.2 slave
>
>
> Regards,
>
> -Humayun.
>
>
> On 22 December 2011 15:47, Uma Maheswara Rao G <maheswara@huawei.com
> <ma...@huawei.com>> wrote:
> Hey Humayun,
>
> To solve the too many fetch failures problem, you should configure host
> mapping correctly.
> Each tasktracker should be able to ping from each other.
>
> Regards,
> Uma
> ________________________________________
> From: Humayun kabir [humayun0156@gmail.com<ma...@gmail.com>]
> Sent: Thursday, December 22, 2011 2:54 PM
> To: common-user@hadoop.apache.org<ma...@hadoop.apache.org>
> Subject: Hadoop configuration
>
> someone please help me to configure hadoop such as core-site.xml,
> hdfs-site.xml, mapred-site.xml etc.
> please provide some example. it is badly needed. because i run in a 2 node
> cluster. when i run the wordcount example then it gives the result too
> mutch fetch failure.
>
>
RE: Hadoop configuration
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
Hi Humayun ,
Lets assume you have JT, TT1, TT2, TT3
Now you should configure the \etc\hosts like below examle
10.18.xx.1 JT
10.18.xx.2 TT1
10.18.xx.3 TT2
10.18.xx.4 TT3
Configure the same set in all the machines, so that all task trackers can talk each other with hostnames correctly. Also pls remove some entries from your files
127.0.0.1 localhost.localdomain localhost
127.0.1.1 humayun
I have seen others already suggested many links for the regular configuration items. Hope you might clear about them.
hope it will help...
Regards,
Uma
________________________________
From: Humayun kabir [humayun0156@gmail.com]
Sent: Thursday, December 22, 2011 10:34 PM
To: common-user@hadoop.apache.org; Uma Maheswara Rao G
Subject: Re: Hadoop configuration
Hello Uma,
Thanks for your cordial and quick reply. It would be great if you explain what you suggested to do. Right now we are running on following
configuration.
We are using hadoop on virtual box. when it is a single node then it works fine for big dataset larger than the default block size. but in case of multinode cluster (2 nodes) we are facing some problems. We are able to ping both "Master->Slave" and "Slave->Master".
Like when the input dataset is smaller than the default block size(64 MB) then it works fine. but when the input dataset is larger than the default block size then it shows ‘too much fetch failure’ in reduce state.
here is the output link
http://paste.ubuntu.com/707517/
this is our /etc/hosts file
192.168.60.147 humayun # Added by NetworkManager
127.0.0.1 localhost.localdomain localhost
::1 humayun localhost6.localdomain6 localhost6
127.0.1.1 humayun
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
192.168.60.1 master
192.168.60.2 slave
Regards,
-Humayun.
On 22 December 2011 15:47, Uma Maheswara Rao G <ma...@huawei.com>> wrote:
Hey Humayun,
To solve the too many fetch failures problem, you should configure host mapping correctly.
Each tasktracker should be able to ping from each other.
Regards,
Uma
________________________________________
From: Humayun kabir [humayun0156@gmail.com<ma...@gmail.com>]
Sent: Thursday, December 22, 2011 2:54 PM
To: common-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Hadoop configuration
someone please help me to configure hadoop such as core-site.xml,
hdfs-site.xml, mapred-site.xml etc.
please provide some example. it is badly needed. because i run in a 2 node
cluster. when i run the wordcount example then it gives the result too
mutch fetch failure.
Re: Hadoop configuration
Posted by Humayun kabir <hu...@gmail.com>.
Hello Uma,
Thanks for your cordial and quick reply. It would be great if you explain
what you suggested to do. Right now we are running on following
configuration.
We are using hadoop on virtual box. when it is a single node then it works
fine for big dataset larger than the default block size. but in case of
multinode cluster (2 nodes) we are facing some problems. We are able to
ping both *"Master->Slave"* and *"Slave->Master"*.
Like when the input dataset is smaller than the default block size(64 MB)
then it works fine. but when the input dataset is larger than the default
block size then it shows ‘too much fetch failure’ in reduce state.
here is the output link
http://paste.ubuntu.com/707517/
this is our /etc/hosts file
192.168.60.147 humayun # Added by NetworkManager
127.0.0.1 localhost.localdomain localhost
::1 humayun localhost6.localdomain6 localhost6
127.0.1.1 humayun
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
192.168.60.1 master
192.168.60.2 slave
Regards,
-Humayun.
On 22 December 2011 15:47, Uma Maheswara Rao G <ma...@huawei.com> wrote:
> Hey Humayun,
>
> To solve the too many fetch failures problem, you should configure host
> mapping correctly.
> Each tasktracker should be able to ping from each other.
>
> Regards,
> Uma
> ________________________________________
> From: Humayun kabir [humayun0156@gmail.com]
> Sent: Thursday, December 22, 2011 2:54 PM
> To: common-user@hadoop.apache.org
> Subject: Hadoop configuration
>
> someone please help me to configure hadoop such as core-site.xml,
> hdfs-site.xml, mapred-site.xml etc.
> please provide some example. it is badly needed. because i run in a 2 node
> cluster. when i run the wordcount example then it gives the result too
> mutch fetch failure.
>
RE: Hadoop configuration
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
Hey Humayun,
To solve the too many fetch failures problem, you should configure host mapping correctly.
Each tasktracker should be able to ping from each other.
Regards,
Uma
________________________________________
From: Humayun kabir [humayun0156@gmail.com]
Sent: Thursday, December 22, 2011 2:54 PM
To: common-user@hadoop.apache.org
Subject: Hadoop configuration
someone please help me to configure hadoop such as core-site.xml,
hdfs-site.xml, mapred-site.xml etc.
please provide some example. it is badly needed. because i run in a 2 node
cluster. when i run the wordcount example then it gives the result too
mutch fetch failure.
Re: Hadoop configuration
Posted by Michael Segel <mi...@hotmail.com>.
Class project due?
Sorry, second set of questions on setting up a 2 node cluster...
Sent from my iPhone
On Dec 22, 2011, at 3:25 AM, "Humayun kabir" <hu...@gmail.com> wrote:
> someone please help me to configure hadoop such as core-site.xml,
> hdfs-site.xml, mapred-site.xml etc.
> please provide some example. it is badly needed. because i run in a 2 node
> cluster. when i run the wordcount example then it gives the result too
> mutch fetch failure.