You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Thibaut Britz <t....@netbreeze.ch> on 2007/12/11 16:51:28 UTC
public Amazon EC2 hadoop images for larger instances (m1.large and
m1.xlarge)
Hi,
The current public images only work on the smaller instances.
It would be very helpful (save me some time) if someone would be so kind
create or publish their hadoop image.
Thibaut
--
View this message in context: http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14276807.html
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: public Amazon EC2 hadoop images for larger instances (m1.large and m1.xlarge)
Posted by Tom White <to...@gmail.com>.
Yes - of course the different architecture means the existing image
won't work! I've created
https://issues.apache.org/jira/browse/HADOOP-2411 to track this issue.
See this thread for how to take advantage of extra CPUs:
http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg02377.html.
Tom
On 12/12/2007, Thibaut Britz <t....@netbreeze.ch> wrote:
>
> Hi Tom,
>
> I got the following error "Client.InvalidParameterValue: The requested
> instance type does not agree with the architecture specified in the AMI
> manifest."
> During meantime, I built a private AMI from the 64 bit version of fedora
> with hadoop version 14.4.
>
> What do I have to do to take advantages of the extra CPUS and the Ram (I
> only increased the mapred.child.java.opts)?
>
>
> My map/reduce code is comparable to wordcount (7413862 combine input pairs,
> 122475 reduce input pairs) and doesn't make a lot of calculations, but still
> takes over 30 seconds to complete, while the cpu is displayed to be idle
> about 99% of the time in top. (If I don't collect pairs in the map function,
> the job finished in 3 seconds (just to test how long it takes for my
> RecordReader to deliver the key/value pairs)). Also increasing the mappers
> dramatically reduces performances. (The above is measured with 2 mappers
> only on one x1.large instance). Any ideas on what causes the performance to
> be so low?
>
>
>
>
> Tom White wrote:
> >
> > Hi Thibaut,
> >
> > Do you know why the existing hadoop images don't work with larger
> > instance types? What's the error message you're getting?
> >
> > It should be relatively easy to change the launch-hadoop-cluster
> > script to specify the instance type. Also, there's some work to be
> > done to configure Hadoop on larger images to take advantage of the
> > extra CPUs and memory.
> >
> > BTW I just published a 0.15.1 AMI.
> >
> > Tom
> >
> > On 11/12/2007, Thibaut Britz <t....@netbreeze.ch> wrote:
> >>
> >> Hi,
> >>
> >> The current public images only work on the smaller instances.
> >> It would be very helpful (save me some time) if someone would be so kind
> >> create or publish their hadoop image.
> >>
> >> Thibaut
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14276807.html
> >> Sent from the Hadoop Users mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Blog: http://problemsworthyofattack.blogspot.com/
> >
> >
>
> --
> View this message in context: http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14296045.html
> Sent from the Hadoop Users mailing list archive at Nabble.com.
>
>
--
Blog: http://problemsworthyofattack.blogspot.com/
Re: public Amazon EC2 hadoop images for larger instances (m1.large
and m1.xlarge)
Posted by Thibaut Britz <t....@netbreeze.ch>.
Hi Tom,
I got the following error "Client.InvalidParameterValue: The requested
instance type does not agree with the architecture specified in the AMI
manifest."
During meantime, I built a private AMI from the 64 bit version of fedora
with hadoop version 14.4.
What do I have to do to take advantages of the extra CPUS and the Ram (I
only increased the mapred.child.java.opts)?
My map/reduce code is comparable to wordcount (7413862 combine input pairs,
122475 reduce input pairs) and doesn't make a lot of calculations, but still
takes over 30 seconds to complete, while the cpu is displayed to be idle
about 99% of the time in top. (If I don't collect pairs in the map function,
the job finished in 3 seconds (just to test how long it takes for my
RecordReader to deliver the key/value pairs)). Also increasing the mappers
dramatically reduces performances. (The above is measured with 2 mappers
only on one x1.large instance). Any ideas on what causes the performance to
be so low?
Tom White wrote:
>
> Hi Thibaut,
>
> Do you know why the existing hadoop images don't work with larger
> instance types? What's the error message you're getting?
>
> It should be relatively easy to change the launch-hadoop-cluster
> script to specify the instance type. Also, there's some work to be
> done to configure Hadoop on larger images to take advantage of the
> extra CPUs and memory.
>
> BTW I just published a 0.15.1 AMI.
>
> Tom
>
> On 11/12/2007, Thibaut Britz <t....@netbreeze.ch> wrote:
>>
>> Hi,
>>
>> The current public images only work on the smaller instances.
>> It would be very helpful (save me some time) if someone would be so kind
>> create or publish their hadoop image.
>>
>> Thibaut
>>
>> --
>> View this message in context:
>> http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14276807.html
>> Sent from the Hadoop Users mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Blog: http://problemsworthyofattack.blogspot.com/
>
>
--
View this message in context: http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14296045.html
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: public Amazon EC2 hadoop images for larger instances (m1.large and m1.xlarge)
Posted by je...@gmail.com.
The larger images are all 64 bit, so the 32 bit binaries would be none too happy
J
Sent from my BlackBerry device on the Rogers Wireless Network
-----Original Message-----
From: "Tom White" <to...@gmail.com>
Date: Wed, 12 Dec 2007 14:12:28
To:hadoop-user@lucene.apache.org
Subject: Re: public Amazon EC2 hadoop images for larger instances (m1.large and m1.xlarge)
Hi Thibaut,
Do you know why the existing hadoop images don't work with larger
instance types? What's the error message you're getting?
It should be relatively easy to change the launch-hadoop-cluster
script to specify the instance type. Also, there's some work to be
done to configure Hadoop on larger images to take advantage of the
extra CPUs and memory.
BTW I just published a 0.15.1 AMI.
Tom
On 11/12/2007, Thibaut Britz <t....@netbreeze.ch> wrote:
>
> Hi,
>
> The current public images only work on the smaller instances.
> It would be very helpful (save me some time) if someone would be so kind
> create or publish their hadoop image.
>
> Thibaut
>
> --
> View this message in context: http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14276807.html
> Sent from the Hadoop Users mailing list archive at Nabble.com.
>
>
--
Blog: http://problemsworthyofattack.blogspot.com/
Re: public Amazon EC2 hadoop images for larger instances (m1.large and m1.xlarge)
Posted by Tom White <to...@gmail.com>.
Hi Thibaut,
Do you know why the existing hadoop images don't work with larger
instance types? What's the error message you're getting?
It should be relatively easy to change the launch-hadoop-cluster
script to specify the instance type. Also, there's some work to be
done to configure Hadoop on larger images to take advantage of the
extra CPUs and memory.
BTW I just published a 0.15.1 AMI.
Tom
On 11/12/2007, Thibaut Britz <t....@netbreeze.ch> wrote:
>
> Hi,
>
> The current public images only work on the smaller instances.
> It would be very helpful (save me some time) if someone would be so kind
> create or publish their hadoop image.
>
> Thibaut
>
> --
> View this message in context: http://www.nabble.com/public-Amazon-EC2-hadoop-images-for-larger-instances-%28m1.large-and-m1.xlarge%29-tp14276807p14276807.html
> Sent from the Hadoop Users mailing list archive at Nabble.com.
>
>
--
Blog: http://problemsworthyofattack.blogspot.com/