You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Houston King <ho...@gmail.com> on 2014/07/31 20:32:22 UTC
Setting Up First Hadoop / Yarn Cluster
Hey Everyone,
I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and I've
run into some problems that I'm having a heck of a time debugging.
I've been following the guide posted at
http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/ to
setup the cluster. I've gotten through the guide, but, when I attempt to
run either the wordcount, pi, or randomwriter examples most / all my tasks
end up failing:
14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
attempt_1406829336833_0002_m_000000_0, Status : FAILED
14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
attempt_1406829336833_0002_m_000000_1, Status : FAILED
14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
attempt_1406829336833_0002_m_000000_2, Status : FAILED
14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
with state FAILED due to: Task failed task_1406829336833_0002_m_000000
I've been trying to figure out if I have a configuration problem or where
in the logfiles the problem is described, but without much luck. At this
point, I'm looking for any help I can get to get this cluster going. I
appreciate any and all suggestions!
Thanks
~Houston King
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Houston King <ho...@gmail.com>.
I found what was causing trouble (which it looks like others have seen as
well):
Ubuntu (and maybe other distros?) has 127.0.1.1 as a loopback address to
the node's hostname in /etc/hosts. Removing this line resolved some
connection issues my nodes were having.
~Houston King
On Thu, Jul 31, 2014 at 3:39 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> Probably permission issue.
>
>
> On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
> wrote:
>
>> Hey Everyone,
>>
>> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
>> I've run into some problems that I'm having a heck of a time debugging.
>>
>> I've been following the guide posted at
>> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
>> to setup the cluster. I've gotten through the guide, but, when I attempt
>> to run either the wordcount, pi, or randomwriter examples most / all my
>> tasks end up failing:
>>
>> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
>> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_0, Status : FAILED
>> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_1, Status : FAILED
>> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_2, Status : FAILED
>> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
>> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
>> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>>
>> I've been trying to figure out if I have a configuration problem or where
>> in the logfiles the problem is described, but without much luck. At this
>> point, I'm looking for any help I can get to get this cluster going. I
>> appreciate any and all suggestions!
>>
>> Thanks
>> ~Houston King
>>
>
>
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Houston King <ho...@gmail.com>.
I found what was causing trouble (which it looks like others have seen as
well):
Ubuntu (and maybe other distros?) has 127.0.1.1 as a loopback address to
the node's hostname in /etc/hosts. Removing this line resolved some
connection issues my nodes were having.
~Houston King
On Thu, Jul 31, 2014 at 3:39 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> Probably permission issue.
>
>
> On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
> wrote:
>
>> Hey Everyone,
>>
>> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
>> I've run into some problems that I'm having a heck of a time debugging.
>>
>> I've been following the guide posted at
>> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
>> to setup the cluster. I've gotten through the guide, but, when I attempt
>> to run either the wordcount, pi, or randomwriter examples most / all my
>> tasks end up failing:
>>
>> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
>> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_0, Status : FAILED
>> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_1, Status : FAILED
>> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_2, Status : FAILED
>> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
>> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
>> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>>
>> I've been trying to figure out if I have a configuration problem or where
>> in the logfiles the problem is described, but without much luck. At this
>> point, I'm looking for any help I can get to get this cluster going. I
>> appreciate any and all suggestions!
>>
>> Thanks
>> ~Houston King
>>
>
>
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Houston King <ho...@gmail.com>.
I found what was causing trouble (which it looks like others have seen as
well):
Ubuntu (and maybe other distros?) has 127.0.1.1 as a loopback address to
the node's hostname in /etc/hosts. Removing this line resolved some
connection issues my nodes were having.
~Houston King
On Thu, Jul 31, 2014 at 3:39 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> Probably permission issue.
>
>
> On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
> wrote:
>
>> Hey Everyone,
>>
>> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
>> I've run into some problems that I'm having a heck of a time debugging.
>>
>> I've been following the guide posted at
>> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
>> to setup the cluster. I've gotten through the guide, but, when I attempt
>> to run either the wordcount, pi, or randomwriter examples most / all my
>> tasks end up failing:
>>
>> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
>> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_0, Status : FAILED
>> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_1, Status : FAILED
>> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_2, Status : FAILED
>> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
>> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
>> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>>
>> I've been trying to figure out if I have a configuration problem or where
>> in the logfiles the problem is described, but without much luck. At this
>> point, I'm looking for any help I can get to get this cluster going. I
>> appreciate any and all suggestions!
>>
>> Thanks
>> ~Houston King
>>
>
>
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Houston King <ho...@gmail.com>.
I found what was causing trouble (which it looks like others have seen as
well):
Ubuntu (and maybe other distros?) has 127.0.1.1 as a loopback address to
the node's hostname in /etc/hosts. Removing this line resolved some
connection issues my nodes were having.
~Houston King
On Thu, Jul 31, 2014 at 3:39 PM, Alexander Pivovarov <ap...@gmail.com>
wrote:
> Probably permission issue.
>
>
> On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
> wrote:
>
>> Hey Everyone,
>>
>> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
>> I've run into some problems that I'm having a heck of a time debugging.
>>
>> I've been following the guide posted at
>> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
>> to setup the cluster. I've gotten through the guide, but, when I attempt
>> to run either the wordcount, pi, or randomwriter examples most / all my
>> tasks end up failing:
>>
>> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
>> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_0, Status : FAILED
>> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_1, Status : FAILED
>> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
>> attempt_1406829336833_0002_m_000000_2, Status : FAILED
>> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
>> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
>> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>>
>> I've been trying to figure out if I have a configuration problem or where
>> in the logfiles the problem is described, but without much luck. At this
>> point, I'm looking for any help I can get to get this cluster going. I
>> appreciate any and all suggestions!
>>
>> Thanks
>> ~Houston King
>>
>
>
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Alexander Pivovarov <ap...@gmail.com>.
Probably permission issue.
On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
wrote:
> Hey Everyone,
>
> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
> I've run into some problems that I'm having a heck of a time debugging.
>
> I've been following the guide posted at
> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
> to setup the cluster. I've gotten through the guide, but, when I attempt
> to run either the wordcount, pi, or randomwriter examples most / all my
> tasks end up failing:
>
> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_0, Status : FAILED
> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_1, Status : FAILED
> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_2, Status : FAILED
> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>
> I've been trying to figure out if I have a configuration problem or where
> in the logfiles the problem is described, but without much luck. At this
> point, I'm looking for any help I can get to get this cluster going. I
> appreciate any and all suggestions!
>
> Thanks
> ~Houston King
>
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Alexander Pivovarov <ap...@gmail.com>.
Probably permission issue.
On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
wrote:
> Hey Everyone,
>
> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
> I've run into some problems that I'm having a heck of a time debugging.
>
> I've been following the guide posted at
> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
> to setup the cluster. I've gotten through the guide, but, when I attempt
> to run either the wordcount, pi, or randomwriter examples most / all my
> tasks end up failing:
>
> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_0, Status : FAILED
> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_1, Status : FAILED
> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_2, Status : FAILED
> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>
> I've been trying to figure out if I have a configuration problem or where
> in the logfiles the problem is described, but without much luck. At this
> point, I'm looking for any help I can get to get this cluster going. I
> appreciate any and all suggestions!
>
> Thanks
> ~Houston King
>
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Alexander Pivovarov <ap...@gmail.com>.
Probably permission issue.
On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
wrote:
> Hey Everyone,
>
> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
> I've run into some problems that I'm having a heck of a time debugging.
>
> I've been following the guide posted at
> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
> to setup the cluster. I've gotten through the guide, but, when I attempt
> to run either the wordcount, pi, or randomwriter examples most / all my
> tasks end up failing:
>
> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_0, Status : FAILED
> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_1, Status : FAILED
> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_2, Status : FAILED
> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>
> I've been trying to figure out if I have a configuration problem or where
> in the logfiles the problem is described, but without much luck. At this
> point, I'm looking for any help I can get to get this cluster going. I
> appreciate any and all suggestions!
>
> Thanks
> ~Houston King
>
Re: Setting Up First Hadoop / Yarn Cluster
Posted by Alexander Pivovarov <ap...@gmail.com>.
Probably permission issue.
On Thu, Jul 31, 2014 at 11:32 AM, Houston King <ho...@gmail.com>
wrote:
> Hey Everyone,
>
> I'm a noob working to setup my first 13 node Hadoop 2.4.0 cluster, and
> I've run into some problems that I'm having a heck of a time debugging.
>
> I've been following the guide posted at
> http://www.implementhadoop.com/install-hadoop-2-4-0-multi-node-cluster/
> to setup the cluster. I've gotten through the guide, but, when I attempt
> to run either the wordcount, pi, or randomwriter examples most / all my
> tasks end up failing:
>
> 14/07/31 12:23:14 INFO mapreduce.Job: map 0% reduce 0%
> 14/07/31 12:23:28 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_0, Status : FAILED
> 14/07/31 12:23:42 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_1, Status : FAILED
> 14/07/31 12:23:56 INFO mapreduce.Job: Task Id :
> attempt_1406829336833_0002_m_000000_2, Status : FAILED
> 14/07/31 12:24:12 INFO mapreduce.Job: map 100% reduce 100%
> 14/07/31 12:24:13 INFO mapreduce.Job: Job job_1406829336833_0002 failed
> with state FAILED due to: Task failed task_1406829336833_0002_m_000000
>
> I've been trying to figure out if I have a configuration problem or where
> in the logfiles the problem is described, but without much luck. At this
> point, I'm looking for any help I can get to get this cluster going. I
> appreciate any and all suggestions!
>
> Thanks
> ~Houston King
>