You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Jiamin Lu <ji...@googlemail.com> on 2011/11/24 16:40:29 UTC
How many real cores in large type instances on EC2 ?
Hi, all
I am using the Amazon EC2, with their large instances.
Amazon claims these large type instances have 4 EC2 Compute units (2
virtual cores with 2 EC2 Compute Units each).
But according to my observation, it seems like they only have two cores.
I checked the /proc/cpuinfo, which shows there are only two processors,
I also used the *top* command, and it also says only two cpu there.
Can someone tell me actually how many cores are contained inside these
large instances?
Did I misunderstand these terms that Amazon talks about ??
Thanks
Jiamin Lu
Re: How many real cores in large type instances on EC2 ?
Posted by Jiamin Lu <ji...@googlemail.com>.
Hi, all
Thanks for the help that I got from this mailing list, the link given by
Harsh is very helpful, thank you very much. Here is my conclusion.
I think Amazon didn't explain this question well, and some concepts are
also confused.
First, the core that I understand is a processor chip inside a multi-core
CPU, but in Amazon, a core means a physical CPU which contains several
ECUs. That's why they say the cores in High-CPU Medium Instance contain 2.5
ECU each. I agree to Rebert, that the ECU is equal to the concept of
processing power, and the power of each ECU is similar with a 1.0 ~1.2 GHz
2007 Opteron or 2007 Xeon Processor.
This is really confuse, especially when people find the EC2 small instance
contains a 2.6GHz cpu, and the large instance only contains two 2.0GHz cpu,
though Amazon claims that the large one should be 4 times as powerful as
the small one. The reason here is that the 2.6GHz cpu in small instance is
a dual-core processor, contains 2 ECU, and it only can use the half power
of it, i.e., one ECU. But in the large instance, we can use the full power
of these processors, i.e. four ECUs together.
Another confusing thing is that in my own computer, which has a 6-core
processor, I can see clearly that it has 6 items listed in /proc/cpuinfo,
and I can see the usage of each core in top window. But in EC2, I can only
see 2 physical cpu information there, even though it's true has 4
cores(ECU) in total. The only difference is that the speed of each core is
800MHz in my computer, and it's 2004MHz in EC2 instance.
I did some experiment by running 4 similar processes in parallel on one EC2
large instance, then I find that each process can almost evenly take 50%
power of each cpu. I think that's means we do can fully use these 4 ECUs to
process tasks independently.
Thanks again, and hope this conclusion can help the other people who are
also interested about EC2 instances, which I think is really good place for
building a temporary large scale cluster.
On Fri, Nov 25, 2011 at 9:56 AM, Robert Hafner <te...@tedivm.com> wrote:
>
> You're looking at two different things. The number of cores is one thing,
> and the power of each core another. Number of cores is pretty straight
> forward, and the power is relative to that of a small instance. You've got
> the answer right there in your email- there are two virtual cores
> of two compute units each, so you will only see two cores. Those cores are
> twice as powerful as the typical core (hence the two compute units), but
> they are still only going to show two cores.
>
> This does have a bit of an effect on processing jobs. If you need more CPU
> power behind your tasks than a large instance will be nice, but if your
> bottleneck is something else (memory, disk i/o, s3 access, etc) then you'll
> probably be optimizing for the wrong problem. In my experience it's better
> to have four small instances running in a job than one large instance, even
> though the cost is equivalent. If you're using spot instances- which are a
> huge budget saver- then the small instances make even more sense, since
> lost machines will only result in one lost task instead of four.
>
> Robert
>
>
> On Nov 24, 2011, at 7:40 AM , Jiamin Lu wrote:
>
> Hi, all
>
> I am using the Amazon EC2, with their large instances.
> Amazon claims these large type instances have 4 EC2 Compute units (2
> virtual cores with 2 EC2 Compute Units each).
> But according to my observation, it seems like they only have two cores.
>
> I checked the /proc/cpuinfo, which shows there are only two processors,
> I also used the *top* command, and it also says only two cpu there.
>
> Can someone tell me actually how many cores are contained inside these
> large instances?
> Did I misunderstand these terms that Amazon talks about ??
>
> Thanks
>
> Jiamin Lu
>
>
>
--
Jiamin Lu
Fernuniversität Hagen
D-58084 Hagen
GERMANY
Phone: +49-2331-987-4276
Email: Jiamin.Luu@gmail.com
Re: How many real cores in large type instances on EC2 ?
Posted by Robert Hafner <te...@tedivm.com>.
You're looking at two different things. The number of cores is one thing, and the power of each core another. Number of cores is pretty straight forward, and the power is relative to that of a small instance. You've got the answer right there in your email- there are two virtual cores of two compute units each, so you will only see two cores. Those cores are twice as powerful as the typical core (hence the two compute units), but they are still only going to show two cores.
This does have a bit of an effect on processing jobs. If you need more CPU power behind your tasks than a large instance will be nice, but if your bottleneck is something else (memory, disk i/o, s3 access, etc) then you'll probably be optimizing for the wrong problem. In my experience it's better to have four small instances running in a job than one large instance, even though the cost is equivalent. If you're using spot instances- which are a huge budget saver- then the small instances make even more sense, since lost machines will only result in one lost task instead of four.
Robert
On Nov 24, 2011, at 7:40 AM , Jiamin Lu wrote:
> Hi, all
>
> I am using the Amazon EC2, with their large instances.
> Amazon claims these large type instances have 4 EC2 Compute units (2 virtual cores with 2 EC2 Compute Units each).
> But according to my observation, it seems like they only have two cores.
>
> I checked the /proc/cpuinfo, which shows there are only two processors,
> I also used the top command, and it also says only two cpu there.
>
> Can someone tell me actually how many cores are contained inside these large instances?
> Did I misunderstand these terms that Amazon talks about ??
>
> Thanks
>
> Jiamin Lu
>
Re: Customized TextInputFormat
Posted by be...@gmail.com.
Hi Riadh
I'm not sure on the customization to achieve that goal, but it should be possible. But if your requirement is to output the same line n number of times as the map output, you don't need to go that way. Just call the outputCollector.collect() or context.write() within a loop in your map method. If you could post in your requirement then may be we can help you better.
Regards
Bejoy K S
-----Original Message-----
From: Mohamed Riadh Trad <Mo...@inria.fr>
Date: Thu, 24 Nov 2011 18:20:26
To: <ma...@hadoop.apache.org>
Reply-To: mapreduce-user@hadoop.apache.org
Subject: Customized TextInputFormat
Hi All,
Is possible to customize the TextInputFormat in a way that it return the same line n times but with different keys, as for example it returns the same line but with different start positions?
Bests,
Trad Mohamed Riadh, M.Sc, Ing.
PhD. student
INRIA-TELECOM PARISTECH - ENPC School of International Management
Office: 11-15
Phone: (33)-1 39 63 59 33
Fax: (33)-1 39 63 56 74
Email: riadh.trad@inria.fr
Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/
Customized TextInputFormat
Posted by Mohamed Riadh Trad <Mo...@inria.fr>.
Hi All,
Is possible to customize the TextInputFormat in a way that it return the same line n times but with different keys, as for example it returns the same line but with different start positions?
Bests,
Trad Mohamed Riadh, M.Sc, Ing.
PhD. student
INRIA-TELECOM PARISTECH - ENPC School of International Management
Office: 11-15
Phone: (33)-1 39 63 59 33
Fax: (33)-1 39 63 56 74
Email: riadh.trad@inria.fr
Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/
Re: How many real cores in large type instances on EC2 ?
Posted by Harsh J <ha...@cloudera.com>.
Hi Mohamed,
[Always better to start your own thread instead of hijacking another :)]
Yes, this is possible. You just need a custom RecordReader implemented
within your subclass of TextInputFormat that performs the logic you
require.
On Thu, Nov 24, 2011 at 10:49 PM, Mohamed Riadh Trad
<Mo...@inria.fr> wrote:
> Hi All,
> Is possible to customize the TextInputFormat in a way that it return the
> same line n times but with different keys, as for example it returns the
> same line but with different start positions?
> Bests,
>
> Trad Mohamed Riadh, M.Sc, Ing.
> PhD. student
> INRIA-TELECOM PARISTECH - ENPC School of International Management
> Office: 11-15
> Phone: (33)-1 39 63 59 33
> Fax: (33)-1 39 63 56 74
> Email: riadh.trad@inria.fr
> Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/
--
Harsh J
Re: How many real cores in large type instances on EC2 ?
Posted by Mohamed Riadh Trad <Mo...@inria.fr>.
Hi All,
Is possible to customize the TextInputFormat in a way that it return the same line n times but with different keys, as for example it returns the same line but with different start positions?
Bests,
Trad Mohamed Riadh, M.Sc, Ing.
PhD. student
INRIA-TELECOM PARISTECH - ENPC School of International Management
Office: 11-15
Phone: (33)-1 39 63 59 33
Fax: (33)-1 39 63 56 74
Email: riadh.trad@inria.fr
Home page: http://www-rocq.inria.fr/who/Mohamed.Trad/
Re: How many real cores in large type instances on EC2 ?
Posted by Harsh J <ha...@cloudera.com>.
Jiamin,
Agree it is. I just wanted you to head to the right source for the proper answer :)
Here are the cpuinfos of each instance, so be assured you're not being given anything lower: http://www.cloudiquity.com/2009/01/amazon-ec2-instances-and-cpuinfo/
On 25-Nov-2011, at 12:01 PM, Jiamin Lu wrote:
> Dear Harsh,
>
> Thanks for your reply and suggestion.
> I will read your recommendation carefully.
> If I find something important, I will left my answer to this thread too, since I think EC2 is quite important in parallel processing.
>
> Thanks
>
>
> On Thu, Nov 24, 2011 at 6:31 PM, Harsh J <ha...@cloudera.com> wrote:
> Jiamin,
>
> Does AMZN not carry a forum of their own for these questions? They'd
> be the best to ask really.
>
> I do not know what an "EC2 Compute Unit" means, but the page clearly
> says two virtual cores. Perhaps this may help understand:
> http://gevaperry.typepad.com/main/2009/03/figuring-out-the-roi-of-infrastructureasaservice.html
>
> On Thu, Nov 24, 2011 at 9:10 PM, Jiamin Lu <ji...@googlemail.com> wrote:
> > Hi, all
> >
> > I am using the Amazon EC2, with their large instances.
> > Amazon claims these large type instances have 4 EC2 Compute units (2 virtual
> > cores with 2 EC2 Compute Units each).
> > But according to my observation, it seems like they only have two cores.
> >
> > I checked the /proc/cpuinfo, which shows there are only two processors,
> > I also used the top command, and it also says only two cpu there.
> >
> > Can someone tell me actually how many cores are contained inside these large
> > instances?
> > Did I misunderstand these terms that Amazon talks about ??
> >
> > Thanks
> >
> > Jiamin Lu
> >
> >
>
>
>
> --
> Harsh J
>
>
>
> --
> Jiamin Lu
> Fernuniversität Hagen
> D-58084 Hagen
> GERMANY
>
> Phone: +49-2331-987-4276
> Email: Jiamin.Luu@gmail.com
Re: How many real cores in large type instances on EC2 ?
Posted by Jiamin Lu <ji...@googlemail.com>.
Dear Harsh,
Thanks for your reply and suggestion.
I will read your recommendation carefully.
If I find something important, I will left my answer to this thread too,
since I think EC2 is quite important in parallel processing.
Thanks
On Thu, Nov 24, 2011 at 6:31 PM, Harsh J <ha...@cloudera.com> wrote:
> Jiamin,
>
> Does AMZN not carry a forum of their own for these questions? They'd
> be the best to ask really.
>
> I do not know what an "EC2 Compute Unit" means, but the page clearly
> says two virtual cores. Perhaps this may help understand:
>
> http://gevaperry.typepad.com/main/2009/03/figuring-out-the-roi-of-infrastructureasaservice.html
>
> On Thu, Nov 24, 2011 at 9:10 PM, Jiamin Lu <ji...@googlemail.com>
> wrote:
> > Hi, all
> >
> > I am using the Amazon EC2, with their large instances.
> > Amazon claims these large type instances have 4 EC2 Compute units (2
> virtual
> > cores with 2 EC2 Compute Units each).
> > But according to my observation, it seems like they only have two cores.
> >
> > I checked the /proc/cpuinfo, which shows there are only two processors,
> > I also used the top command, and it also says only two cpu there.
> >
> > Can someone tell me actually how many cores are contained inside these
> large
> > instances?
> > Did I misunderstand these terms that Amazon talks about ??
> >
> > Thanks
> >
> > Jiamin Lu
> >
> >
>
>
>
> --
> Harsh J
>
--
Jiamin Lu
Fernuniversität Hagen
D-58084 Hagen
GERMANY
Phone: +49-2331-987-4276
Email: Jiamin.Luu@gmail.com
Re: How many real cores in large type instances on EC2 ?
Posted by Harsh J <ha...@cloudera.com>.
Jiamin,
Does AMZN not carry a forum of their own for these questions? They'd
be the best to ask really.
I do not know what an "EC2 Compute Unit" means, but the page clearly
says two virtual cores. Perhaps this may help understand:
http://gevaperry.typepad.com/main/2009/03/figuring-out-the-roi-of-infrastructureasaservice.html
On Thu, Nov 24, 2011 at 9:10 PM, Jiamin Lu <ji...@googlemail.com> wrote:
> Hi, all
>
> I am using the Amazon EC2, with their large instances.
> Amazon claims these large type instances have 4 EC2 Compute units (2 virtual
> cores with 2 EC2 Compute Units each).
> But according to my observation, it seems like they only have two cores.
>
> I checked the /proc/cpuinfo, which shows there are only two processors,
> I also used the top command, and it also says only two cpu there.
>
> Can someone tell me actually how many cores are contained inside these large
> instances?
> Did I misunderstand these terms that Amazon talks about ??
>
> Thanks
>
> Jiamin Lu
>
>
--
Harsh J