You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Krish Donald <go...@gmail.com> on 2015/03/05 23:28:54 UTC
AWS Setting for setting up Hadoop cluster
Hi,
I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
using .
And I was using very basic Hadoop services only.
But it is so slow that I am not able to do anything on that.
Hence I would like to try the AWS service now.
Can anybody please help me, which configuration I should use it without
paying at all?
What are the tips you have for AWS ?
Thanks
Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
Advantage of EMR is that you dont have to stay screwing around with
installing hadoop it does all that for you so you are ready to go
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:51, Krish Donald wrote:
> Because I am new to AWS, I would like to explore the free service first and then later I can use EMR.
> Which one is fast in EC2 and free too?
>
> Thanks
>
> On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net> wrote:
>
> I have experience with my full time job using EMR damn thing is quick and cheap. The interesting part is wrapping your head around the concepts. If you need things quickly and fast EMR is the way to go. It spawns up a number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are m3.large nodes which run you 7 cents per hour. to run one years with of data which is about 1.1 billion records from the database it took 50 min from cluster spawn up to completion and shutting down of the cluster.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
Advantage of EMR is that you dont have to stay screwing around with
installing hadoop it does all that for you so you are ready to go
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:51, Krish Donald wrote:
> Because I am new to AWS, I would like to explore the free service first and then later I can use EMR.
> Which one is fast in EC2 and free too?
>
> Thanks
>
> On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net> wrote:
>
> I have experience with my full time job using EMR damn thing is quick and cheap. The interesting part is wrapping your head around the concepts. If you need things quickly and fast EMR is the way to go. It spawns up a number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are m3.large nodes which run you 7 cents per hour. to run one years with of data which is about 1.1 billion records from the database it took 50 min from cluster spawn up to completion and shutting down of the cluster.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
Advantage of EMR is that you dont have to stay screwing around with
installing hadoop it does all that for you so you are ready to go
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:51, Krish Donald wrote:
> Because I am new to AWS, I would like to explore the free service first and then later I can use EMR.
> Which one is fast in EC2 and free too?
>
> Thanks
>
> On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net> wrote:
>
> I have experience with my full time job using EMR damn thing is quick and cheap. The interesting part is wrapping your head around the concepts. If you need things quickly and fast EMR is the way to go. It spawns up a number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are m3.large nodes which run you 7 cents per hour. to run one years with of data which is about 1.1 billion records from the database it took 50 min from cluster spawn up to completion and shutting down of the cluster.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
Advantage of EMR is that you dont have to stay screwing around with
installing hadoop it does all that for you so you are ready to go
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:51, Krish Donald wrote:
> Because I am new to AWS, I would like to explore the free service first and then later I can use EMR.
> Which one is fast in EC2 and free too?
>
> Thanks
>
> On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net> wrote:
>
> I have experience with my full time job using EMR damn thing is quick and cheap. The interesting part is wrapping your head around the concepts. If you need things quickly and fast EMR is the way to go. It spawns up a number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are m3.large nodes which run you 7 cents per hour. to run one years with of data which is about 1.1 billion records from the database it took 50 min from cluster spawn up to completion and shutting down of the cluster.
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Krish Donald <go...@gmail.com>.
Because I am new to AWS, I would like to explore the free service first and
then later I can use EMR.
Which one is fast in EC2 and free too?
Thanks
On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net>
wrote:
> I have experience with my full time job using EMR damn thing is quick
> and cheap. The interesting part is wrapping your head around the concepts.
> If you need things quickly and fast EMR is the way to go. It spawns up a
> number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are
> m3.large nodes which run you 7 cents per hour. to run one years with of
> data which is about 1.1 billion records from the database it took 50 min
> from cluster spawn up to completion and shutting down of the cluster.
>
>
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
>
> You can install Hadoop on Amazon EC2 instances and use the free tier for
> new members but you can also use Amazon EMR which is not free but is up and
> running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
>> using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without
>> paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
>>
>
Re: AWS Setting for setting up Hadoop cluster
Posted by Krish Donald <go...@gmail.com>.
Because I am new to AWS, I would like to explore the free service first and
then later I can use EMR.
Which one is fast in EC2 and free too?
Thanks
On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net>
wrote:
> I have experience with my full time job using EMR damn thing is quick
> and cheap. The interesting part is wrapping your head around the concepts.
> If you need things quickly and fast EMR is the way to go. It spawns up a
> number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are
> m3.large nodes which run you 7 cents per hour. to run one years with of
> data which is about 1.1 billion records from the database it took 50 min
> from cluster spawn up to completion and shutting down of the cluster.
>
>
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
>
> You can install Hadoop on Amazon EC2 instances and use the free tier for
> new members but you can also use Amazon EMR which is not free but is up and
> running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
>> using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without
>> paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
>>
>
Re: AWS Setting for setting up Hadoop cluster
Posted by Krish Donald <go...@gmail.com>.
Because I am new to AWS, I would like to explore the free service first and
then later I can use EMR.
Which one is fast in EC2 and free too?
Thanks
On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net>
wrote:
> I have experience with my full time job using EMR damn thing is quick
> and cheap. The interesting part is wrapping your head around the concepts.
> If you need things quickly and fast EMR is the way to go. It spawns up a
> number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are
> m3.large nodes which run you 7 cents per hour. to run one years with of
> data which is about 1.1 billion records from the database it took 50 min
> from cluster spawn up to completion and shutting down of the cluster.
>
>
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
>
> You can install Hadoop on Amazon EC2 instances and use the free tier for
> new members but you can also use Amazon EMR which is not free but is up and
> running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
>> using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without
>> paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
>>
>
Re: AWS Setting for setting up Hadoop cluster
Posted by Krish Donald <go...@gmail.com>.
Because I am new to AWS, I would like to explore the free service first and
then later I can use EMR.
Which one is fast in EC2 and free too?
Thanks
On Thu, Mar 5, 2015 at 2:47 PM, Jonathan Aquilina <ja...@eagleeyet.net>
wrote:
> I have experience with my full time job using EMR damn thing is quick
> and cheap. The interesting part is wrapping your head around the concepts.
> If you need things quickly and fast EMR is the way to go. It spawns up a
> number of ec2 instances
>
> by default you have 1 master and 2 core nodes. The three of them are
> m3.large nodes which run you 7 cents per hour. to run one years with of
> data which is about 1.1 billion records from the database it took 50 min
> from cluster spawn up to completion and shutting down of the cluster.
>
>
>
> ---
> Regards,
> Jonathan Aquilina
> Founder Eagle Eye T
>
> On 2015-03-05 23:41, Dieter De Witte wrote:
>
> You can install Hadoop on Amazon EC2 instances and use the free tier for
> new members but you can also use Amazon EMR which is not free but is up and
> running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
>> using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without
>> paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
>>
>
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I have experience with my full time job using EMR damn thing is quick
and cheap. The interesting part is wrapping your head around the
concepts. If you need things quickly and fast EMR is the way to go. It
spawns up a number of ec2 instances
by default you have 1 master and 2 core nodes. The three of them are
m3.large nodes which run you 7 cents per hour. to run one years with of
data which is about 1.1 billion records from the database it took 50 min
from cluster spawn up to completion and shutting down of the cluster.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I have experience with my full time job using EMR damn thing is quick
and cheap. The interesting part is wrapping your head around the
concepts. If you need things quickly and fast EMR is the way to go. It
spawns up a number of ec2 instances
by default you have 1 master and 2 core nodes. The three of them are
m3.large nodes which run you 7 cents per hour. to run one years with of
data which is about 1.1 billion records from the database it took 50 min
from cluster spawn up to completion and shutting down of the cluster.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I have experience with my full time job using EMR damn thing is quick
and cheap. The interesting part is wrapping your head around the
concepts. If you need things quickly and fast EMR is the way to go. It
spawns up a number of ec2 instances
by default you have 1 master and 2 core nodes. The three of them are
m3.large nodes which run you 7 cents per hour. to run one years with of
data which is about 1.1 billion records from the database it took 50 min
from cluster spawn up to completion and shutting down of the cluster.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Jonathan Aquilina <ja...@eagleeyet.net>.
I have experience with my full time job using EMR damn thing is quick
and cheap. The interesting part is wrapping your head around the
concepts. If you need things quickly and fast EMR is the way to go. It
spawns up a number of ec2 instances
by default you have 1 master and 2 core nodes. The three of them are
m3.large nodes which run you 7 cents per hour. to run one years with of
data which is about 1.1 billion records from the database it took 50 min
from cluster spawn up to completion and shutting down of the cluster.
---
Regards,
Jonathan Aquilina
Founder Eagle Eye T
On 2015-03-05 23:41, Dieter De Witte wrote:
> You can install Hadoop on Amazon EC2 instances and use the free tier for new members but you can also use Amazon EMR which is not free but is up and running in a couple of seconds...
>
> 2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
>
>> Hi,
>>
>> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
>> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was using .
>> And I was using very basic Hadoop services only.
>> But it is so slow that I am not able to do anything on that.
>>
>> Hence I would like to try the AWS service now.
>>
>> Can anybody please help me, which configuration I should use it without paying at all?
>> What are the tips you have for AWS ?
>>
>> Thanks
>> Krish
Re: AWS Setting for setting up Hadoop cluster
Posted by Dieter De Witte <dr...@gmail.com>.
You can install Hadoop on Amazon EC2 instances and use the free tier for
new members but you can also use Amazon EMR which is not free but is up and
running in a couple of seconds...
2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
> using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without
> paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
>
Re: AWS Setting for setting up Hadoop cluster
Posted by Dieter De Witte <dr...@gmail.com>.
You can install Hadoop on Amazon EC2 instances and use the free tier for
new members but you can also use Amazon EMR which is not free but is up and
running in a couple of seconds...
2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
> using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without
> paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
>
Re: AWS Setting for setting up Hadoop cluster
Posted by Dieter De Witte <dr...@gmail.com>.
You can install Hadoop on Amazon EC2 instances and use the free tier for
new members but you can also use Amazon EMR which is not free but is up and
running in a couple of seconds...
2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
> using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without
> paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
>
Re: AWS Setting for setting up Hadoop cluster
Posted by Dieter De Witte <dr...@gmail.com>.
You can install Hadoop on Amazon EC2 instances and use the free tier for
new members but you can also use Amazon EMR which is not free but is up and
running in a couple of seconds...
2015-03-05 23:28 GMT+01:00 Krish Donald <go...@gmail.com>:
> Hi,
>
> I am tired of setting Hadoop cluster using my laptop which has 8GB RAM.
> I tried 2gb for namenode and 1-1 gb for 3 datanoded so total 5gb I was
> using .
> And I was using very basic Hadoop services only.
> But it is so slow that I am not able to do anything on that.
>
> Hence I would like to try the AWS service now.
>
> Can anybody please help me, which configuration I should use it without
> paying at all?
> What are the tips you have for AWS ?
>
> Thanks
> Krish
>