You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Shahnawaz Saifi <sh...@gmail.com> on 2011/09/07 13:18:27 UTC
Hadoop on Ec2
Hi,
I was trying to set-up hadoop/hbase cluster on ec2 which took me few hours
to set-up from scratch on bundled image from s3. I am curious to know, what
is the best way to setting hadoop/hbase cluster on amazon ec2? How do we do
it fast?
Thanks in advance!
regards,
Shah
Re: Hadoop on Ec2
Posted by Shi Yu <sh...@uchicago.edu>.
Interested in this topic. We have experienced plenty of difficulties
running hadoop in Eucalyptus based virtual instance clusters. Typical
issues like
java.net.SocketTimeoutException: 69000 millis timeout while waiting for
channel to be ready for read. ch : java.nio.channels.SocketChannel
kill the whole job. The IO of HDFS based on network storage is very
slow. I am wondering whether Apache Whirr has made any significant
improvement for hadoop implementation in virtual instances like Ec2.
On 9/7/2011 9:58 AM, John Conwell wrote:
> I second that. Whirr is an invaluable resource for automagically spinning
> up resources on EC2
>
> On Wed, Sep 7, 2011 at 4:28 AM, Harsh J<ha...@cloudera.com> wrote:
>
>> You are looking for the Apache Whirr project: http://whirr.apache.org/
>>
>> Here's a great article at Phil Whelan's site that covers getting HBase
>> up in a jiffy on ec2:
>> http://www.philwhln.com/run-the-latest-whirr-and-deploy-hbase-in-minutes
>>
>> On Wed, Sep 7, 2011 at 4:48 PM, Shahnawaz Saifi<sh...@gmail.com>
>> wrote:
>>> Hi,
>>>
>>> I was trying to set-up hadoop/hbase cluster on ec2 which took me few
>> hours
>>> to set-up from scratch on bundled image from s3. I am curious to know,
>> what
>>> is the best way to setting hadoop/hbase cluster on amazon ec2? How do we
>> do
>>> it fast?
>>>
>>> Thanks in advance!
>>>
>>> regards,
>>> Shah
>>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Hadoop on Ec2
Posted by John Conwell <jo...@iamjohn.me>.
I dont know about configuring hbase, but whirr definitely makes spinning up
a hadoop cluster very very easy. Checkout the quickstart guide. As long as
you have your EC2 creds, its very simple to ramp up a hadoop cluster.
http://whirr.apache.org/docs/0.6.0/quick-start-guide.html
On Wed, Sep 7, 2011 at 10:03 AM, Shahnawaz Saifi <sh...@gmail.com>wrote:
> Thanks a lot, I will definitely try this. But there are so many blogs about
> configuring hadoop/hbase and bundling images to s3 bucket. Whirr is faster
> or smoother than this concept?
>
> regards,
> Shah
>
> On Wed, Sep 7, 2011 at 8:28 PM, John Conwell <jo...@iamjohn.me> wrote:
>
> > I second that. Whirr is an invaluable resource for automagically
> spinning
> > up resources on EC2
> >
> > On Wed, Sep 7, 2011 at 4:28 AM, Harsh J <ha...@cloudera.com> wrote:
> >
> > > You are looking for the Apache Whirr project: http://whirr.apache.org/
> > >
> > > Here's a great article at Phil Whelan's site that covers getting HBase
> > > up in a jiffy on ec2:
> > >
> http://www.philwhln.com/run-the-latest-whirr-and-deploy-hbase-in-minutes
> > >
> > > On Wed, Sep 7, 2011 at 4:48 PM, Shahnawaz Saifi <sh...@gmail.com>
> > > wrote:
> > > > Hi,
> > > >
> > > > I was trying to set-up hadoop/hbase cluster on ec2 which took me few
> > > hours
> > > > to set-up from scratch on bundled image from s3. I am curious to
> know,
> > > what
> > > > is the best way to setting hadoop/hbase cluster on amazon ec2? How do
> > we
> > > do
> > > > it fast?
> > > >
> > > > Thanks in advance!
> > > >
> > > > regards,
> > > > Shah
> > > >
> > >
> > >
> > >
> > > --
> > > Harsh J
> > >
> >
> >
> >
> > --
> >
> > Thanks,
> > John C
> >
>
>
>
> --
> Thanks,
> Shah
>
--
Thanks,
John C
Re: Hadoop on Ec2
Posted by Shahnawaz Saifi <sh...@gmail.com>.
Thanks a lot, I will definitely try this. But there are so many blogs about
configuring hadoop/hbase and bundling images to s3 bucket. Whirr is faster
or smoother than this concept?
regards,
Shah
On Wed, Sep 7, 2011 at 8:28 PM, John Conwell <jo...@iamjohn.me> wrote:
> I second that. Whirr is an invaluable resource for automagically spinning
> up resources on EC2
>
> On Wed, Sep 7, 2011 at 4:28 AM, Harsh J <ha...@cloudera.com> wrote:
>
> > You are looking for the Apache Whirr project: http://whirr.apache.org/
> >
> > Here's a great article at Phil Whelan's site that covers getting HBase
> > up in a jiffy on ec2:
> > http://www.philwhln.com/run-the-latest-whirr-and-deploy-hbase-in-minutes
> >
> > On Wed, Sep 7, 2011 at 4:48 PM, Shahnawaz Saifi <sh...@gmail.com>
> > wrote:
> > > Hi,
> > >
> > > I was trying to set-up hadoop/hbase cluster on ec2 which took me few
> > hours
> > > to set-up from scratch on bundled image from s3. I am curious to know,
> > what
> > > is the best way to setting hadoop/hbase cluster on amazon ec2? How do
> we
> > do
> > > it fast?
> > >
> > > Thanks in advance!
> > >
> > > regards,
> > > Shah
> > >
> >
> >
> >
> > --
> > Harsh J
> >
>
>
>
> --
>
> Thanks,
> John C
>
--
Thanks,
Shah
Re: Hadoop on Ec2
Posted by John Conwell <jo...@iamjohn.me>.
I second that. Whirr is an invaluable resource for automagically spinning
up resources on EC2
On Wed, Sep 7, 2011 at 4:28 AM, Harsh J <ha...@cloudera.com> wrote:
> You are looking for the Apache Whirr project: http://whirr.apache.org/
>
> Here's a great article at Phil Whelan's site that covers getting HBase
> up in a jiffy on ec2:
> http://www.philwhln.com/run-the-latest-whirr-and-deploy-hbase-in-minutes
>
> On Wed, Sep 7, 2011 at 4:48 PM, Shahnawaz Saifi <sh...@gmail.com>
> wrote:
> > Hi,
> >
> > I was trying to set-up hadoop/hbase cluster on ec2 which took me few
> hours
> > to set-up from scratch on bundled image from s3. I am curious to know,
> what
> > is the best way to setting hadoop/hbase cluster on amazon ec2? How do we
> do
> > it fast?
> >
> > Thanks in advance!
> >
> > regards,
> > Shah
> >
>
>
>
> --
> Harsh J
>
--
Thanks,
John C
Re: Hadoop on Ec2
Posted by Harsh J <ha...@cloudera.com>.
You are looking for the Apache Whirr project: http://whirr.apache.org/
Here's a great article at Phil Whelan's site that covers getting HBase
up in a jiffy on ec2:
http://www.philwhln.com/run-the-latest-whirr-and-deploy-hbase-in-minutes
On Wed, Sep 7, 2011 at 4:48 PM, Shahnawaz Saifi <sh...@gmail.com> wrote:
> Hi,
>
> I was trying to set-up hadoop/hbase cluster on ec2 which took me few hours
> to set-up from scratch on bundled image from s3. I am curious to know, what
> is the best way to setting hadoop/hbase cluster on amazon ec2? How do we do
> it fast?
>
> Thanks in advance!
>
> regards,
> Shah
>
--
Harsh J