Problem to deploy hadoop in ec2

[Tao You <nw...@gmail.com>]

Hi,
We deployed Hadoop on several datacenters in EC2. We configed
mapred.job.tracker, fs.default.name and slaves by
external hostname(DNS)s. We can start the hadoop service, but the
inter-communication among mapers and reducers
didnot work. When we did wordcount, error occured as following.
"INFO mapred.JobClient: Task Id :
attempt_201009121539_0002_m_000003_1, Status : FAILED
Too many fetch-failures
WARN mapred.JobClient: Error reading task
outputip-10-48-98-34.eu-west-1.compute.internal"
Is there a way to fix this problem?
Thanks,
Tao You

Making optimum use of cores

["Ratner, Alan S (IS)" <Al...@ngc.com>]

I'm running Hadoop 0.20.2 on a cluster of servers running Ubuntu 10.4.
Each server has 2 quad-core Nehalem CPUs for a total of 8 physical cores
running as 16 virtual cores.  Ubuntu's System Monitor displays 16
squiggly lines showing usage of the 16 virtual cores.  We only seem to
be making use of one of the 16 virtual cores on any slave node and even
on the master node only one virtual core is significantly busy at a
time.  Is there a way to make better use of the cores?  Presumably I
could run Hadoop in a VM assigned to each virtual core but I would think
there must be a more elegant solution.

Alan Ratner

Re: Problem to deploy hadoop in ec2

[Tao You <nw...@gmail.com>]

Hi Xin,
Did anybody else meet this problem to deploy hadoop in EC2? I think
some guy may have solved this problem. Could you tell me the detailded
configuration?
Thanks,
Tao

Re: Problem to deploy hadoop in ec2

[Xin Feng <dr...@gmail.com>]

Hi Tao,

Then this will deal more with the network specifications. Since your
nodes are not within the same local network, the conf/slaves file and
/etc/hosts file should be tailored to reflect the actual IP of your
nodes. Besides, the inter-sub-network communication should be properly
configured (I guess maybe you can try talk with your network
administrator?)


Xin



On Mon, Sep 13, 2010 at 11:58 AM, Tao You <nw...@gmail.com> wrote:
> Hi Feng,
> active nodes are 20 in our evironment(5 in each datacenter), and
> hadoop is actually running. when we did wordcount example, mapers and
> reducers can not communicate to each other using external ip(dns) in
> ec2.
> they alwasy tried to communicate by internal ips. internal ips can be
> used to do interactions in the same datacenters, but internal ips
> cannot be used to do interactions between two nodes of different
> datacenters. so, the following error occured.
> "INFO mapred.JobClient: Task Id :
> attempt_201009121539_0002_m_000003_1, Status : FAILED
> Too many fetch-failures
> WARN mapred.JobClient: Error reading task
> outputip-10-48-98-34.eu-west-1.compute.internal"
> Thanks,
> Tao You
>

Re: Problem to deploy hadoop in ec2

[Tao You <nw...@gmail.com>]

Hi Feng,
active nodes are 20 in our evironment(5 in each datacenter), and
hadoop is actually running. when we did wordcount example, mapers and
reducers can not communicate to each other using external ip(dns) in
ec2.
they alwasy tried to communicate by internal ips. internal ips can be
used to do interactions in the same datacenters, but internal ips
cannot be used to do interactions between two nodes of different
datacenters. so, the following error occured.
"INFO mapred.JobClient: Task Id :
attempt_201009121539_0002_m_000003_1, Status : FAILED
Too many fetch-failures
WARN mapred.JobClient: Error reading task
outputip-10-48-98-34.eu-west-1.compute.internal"
Thanks,
Tao You

Re: Problem to deploy hadoop in ec2

[Xin Feng <dr...@gmail.com>]

Hi Tao,

I think the first step is to make sure your hadoop is actually running
and with adequate HDFS space. You can try checking the number of
active nodes of the cluster and the size of the HDFS space in the UI
or via the command line.

Xin



On Mon, Sep 13, 2010 at 11:36 AM, Tao You <nw...@gmail.com> wrote:
> Hi,
> We deployed Hadoop on several datacenters in EC2. We configed
> mapred.job.tracker, fs.default.name and slaves by
> external hostname(DNS)s. We can start the hadoop service, but the
> inter-communication among mapers and reducers
> didnot work. When we did wordcount, error occured as following.
> "INFO mapred.JobClient: Task Id :
> attempt_201009121539_0002_m_000003_1, Status : FAILED
> Too many fetch-failures
> WARN mapred.JobClient: Error reading task
> outputip-10-48-98-34.eu-west-1.compute.internal"
> Is there a way to fix this problem?
> Thanks,
> Tao You
>