You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@whirr.apache.org by Mohammad Tariq <do...@gmail.com> on 2011/12/19 20:42:33 UTC
Creating a Hadoop cluster(Hbase+Hive+pig)
Hello list,
I am totally new to Whirr and just trying to get a proper feel of
it..I was going through the wiki and with the help of documentation
there I was able to create a cluster that has Hadoop and Hbase..Now I
want to create a proper working cluster that has Hadoop, Hbase, Hive
and Pig together.How can I do it?Also I would like to know, whether it
is possible to add new tools(like Chukwa, Sqoop etc) to my existing
Hadoop cluster using Whirr or not.If yes, then what should be the
approach???Many thanks.
Regards,
Mohammad Tariq
Re: Creating a Hadoop cluster(Hbase+Hive+pig)
Posted by Andrei Savu <sa...@gmail.com>.
And a final note:
We are now getting ready to make the 0.7.0 release public (the vote is
done).
Some of the cool new things we are shipping are: support for both Chef &
Puppet, a lot of
service improvements, Ganglia & Mahout as a service + a lot of core
improvements.
-- Andrei Savu / andreisavu.ro
On Mon, Dec 19, 2011 at 11:43 PM, Andrei Savu <sa...@gmail.com> wrote:
> Sounds great!
>
> FYI here is what we are targeting for 0.8.0:
> http://s.apache.org/whirr-0.8.0-roadmap
>
> Let's stay in touch. It would be great if we could improve Whirr on the
> way to match
> your needs.
>
> -- Andrei Savu / andreisavu.ro
>
> On Mon, Dec 19, 2011 at 11:39 PM, Mohammad Tariq <do...@gmail.com>wrote:
>
>> Hi Andrei,
>>
>> First of all I would like to thank you guys for the great work
>> you all have done.Actually, we are trying to set up a Hadoop cluster
>> on Amazon that includes Chukwa, Hbase, Hive and Pig.We'll use this
>> setup to collect data from remote servers that is coming in xml format
>> using Chukwa.And we'll dump this data to HDFS and Hbase.Then using Pig
>> we'll do some analytics and may be some indexing through Solr etc.So,
>> I was trying to figure out the options we have and I found Whirr most
>> appropriate.Till now we were trying everything locally on a cluster of
>> 4 machines created by us, but now we are planning to move to AWS.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Tue, Dec 20, 2011 at 2:37 AM, Andrei Savu <sa...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > And welcome to the community! I am happy to see that you've been able
>> to get
>> > up and
>> > running by following the instructions on the website.
>> >
>> > Hive is still work in
>> > progress: https://issues.apache.org/jira/browse/WHIRR-343 and the
>> > same is true for Pig.
>> >
>> > You can easily add any service you want. Check the following
>> presentation:
>> > http://www.oscon.com/oscon2011/public/schedule/detail/19214
>> > (it explain both how Whirr works and how to write a service from
>> scratch)
>> >
>> > If you want to take the lead on implementing a new service I can provide
>> > as much assistance as needed on the way.
>> >
>> > Can you tell us a bit more about the end product you are building on
>> Whirr?
>> >
>> > Cheers,
>> >
>> > -- Andrei Savu / andreisavu.ro
>> >
>> >
>> > On Mon, Dec 19, 2011 at 9:42 PM, Mohammad Tariq <do...@gmail.com>
>> wrote:
>> >>
>> >> Hello list,
>> >>
>> >> I am totally new to Whirr and just trying to get a proper feel of
>> >> it..I was going through the wiki and with the help of documentation
>> >> there I was able to create a cluster that has Hadoop and Hbase..Now I
>> >> want to create a proper working cluster that has Hadoop, Hbase, Hive
>> >> and Pig together.How can I do it?Also I would like to know, whether it
>> >> is possible to add new tools(like Chukwa, Sqoop etc) to my existing
>> >> Hadoop cluster using Whirr or not.If yes, then what should be the
>> >> approach???Many thanks.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >
>> >
>>
>
>
Re: Creating a Hadoop cluster(Hbase+Hive+pig)
Posted by Mohammad Tariq <do...@gmail.com>.
Thanks a lot Andrei.
Regards,
Mohammad Tariq
On Tue, Dec 20, 2011 at 3:40 AM, Andrei Savu <sa...@gmail.com> wrote:
>
> On Tue, Dec 20, 2011 at 12:04 AM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> One last question, does Whirr require some special ssh configuration.??I
>> am not
>> able to ssh my Hadoop cluster.
>
>
> No special settings are needed. By default it's using the key pair for the
> current user
> located in ~/.ssh/ unless otherwise specified.
>
> "ssh host-ip" should work just fine as long as you are not running Whirr as
> root on
> the local machine.
>
> Also see http://whirr.apache.org/faq.html (How do I log in to a node in the
> cluster?)
Re: Creating a Hadoop cluster(Hbase+Hive+pig)
Posted by Andrei Savu <sa...@gmail.com>.
On Tue, Dec 20, 2011 at 12:04 AM, Mohammad Tariq <do...@gmail.com> wrote:
> One last question, does Whirr require some special ssh configuration.??I
> am not
> able to ssh my Hadoop cluster.
>
No special settings are needed. By default it's using the key pair for the
current user
located in ~/.ssh/ unless otherwise specified.
"ssh host-ip" should work just fine as long as you are not running Whirr as
root on
the local machine.
Also see http://whirr.apache.org/faq.html (How do I log in to a node in the
cluster?)
Re: Creating a Hadoop cluster(Hbase+Hive+pig)
Posted by Mohammad Tariq <do...@gmail.com>.
Sure Andrei, I would love to be be in touch with you guys.And I am
waiting anxiously for the next release.Its really great to see that
you guys are integrating tools like Mahout & Ganglia.I'll try to do as
much as I can in improving Whirr, as I am just a newbie in this
area.Till then I'll try to become familiar with the system.One last
question, does Whirr require some special ssh configuration.??I am not
able to ssh my Hadoop cluster. Thanks once again.
Regards,
Mohammad Tariq
On Tue, Dec 20, 2011 at 3:13 AM, Andrei Savu <sa...@gmail.com> wrote:
> Sounds great!
>
> FYI here is what we are targeting for
> 0.8.0: http://s.apache.org/whirr-0.8.0-roadmap
>
> Let's stay in touch. It would be great if we could improve Whirr on the way
> to match
> your needs.
>
> -- Andrei Savu / andreisavu.ro
>
> On Mon, Dec 19, 2011 at 11:39 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> Hi Andrei,
>>
>> First of all I would like to thank you guys for the great work
>> you all have done.Actually, we are trying to set up a Hadoop cluster
>> on Amazon that includes Chukwa, Hbase, Hive and Pig.We'll use this
>> setup to collect data from remote servers that is coming in xml format
>> using Chukwa.And we'll dump this data to HDFS and Hbase.Then using Pig
>> we'll do some analytics and may be some indexing through Solr etc.So,
>> I was trying to figure out the options we have and I found Whirr most
>> appropriate.Till now we were trying everything locally on a cluster of
>> 4 machines created by us, but now we are planning to move to AWS.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Tue, Dec 20, 2011 at 2:37 AM, Andrei Savu <sa...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > And welcome to the community! I am happy to see that you've been able to
>> > get
>> > up and
>> > running by following the instructions on the website.
>> >
>> > Hive is still work in
>> > progress: https://issues.apache.org/jira/browse/WHIRR-343 and the
>> > same is true for Pig.
>> >
>> > You can easily add any service you want. Check the following
>> > presentation:
>> > http://www.oscon.com/oscon2011/public/schedule/detail/19214
>> > (it explain both how Whirr works and how to write a service from
>> > scratch)
>> >
>> > If you want to take the lead on implementing a new service I can provide
>> > as much assistance as needed on the way.
>> >
>> > Can you tell us a bit more about the end product you are building on
>> > Whirr?
>> >
>> > Cheers,
>> >
>> > -- Andrei Savu / andreisavu.ro
>> >
>> >
>> > On Mon, Dec 19, 2011 at 9:42 PM, Mohammad Tariq <do...@gmail.com>
>> > wrote:
>> >>
>> >> Hello list,
>> >>
>> >> I am totally new to Whirr and just trying to get a proper feel of
>> >> it..I was going through the wiki and with the help of documentation
>> >> there I was able to create a cluster that has Hadoop and Hbase..Now I
>> >> want to create a proper working cluster that has Hadoop, Hbase, Hive
>> >> and Pig together.How can I do it?Also I would like to know, whether it
>> >> is possible to add new tools(like Chukwa, Sqoop etc) to my existing
>> >> Hadoop cluster using Whirr or not.If yes, then what should be the
>> >> approach???Many thanks.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >
>> >
>
>
Re: Creating a Hadoop cluster(Hbase+Hive+pig)
Posted by Andrei Savu <sa...@gmail.com>.
Sounds great!
FYI here is what we are targeting for 0.8.0:
http://s.apache.org/whirr-0.8.0-roadmap
Let's stay in touch. It would be great if we could improve Whirr on the way
to match
your needs.
-- Andrei Savu / andreisavu.ro
On Mon, Dec 19, 2011 at 11:39 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hi Andrei,
>
> First of all I would like to thank you guys for the great work
> you all have done.Actually, we are trying to set up a Hadoop cluster
> on Amazon that includes Chukwa, Hbase, Hive and Pig.We'll use this
> setup to collect data from remote servers that is coming in xml format
> using Chukwa.And we'll dump this data to HDFS and Hbase.Then using Pig
> we'll do some analytics and may be some indexing through Solr etc.So,
> I was trying to figure out the options we have and I found Whirr most
> appropriate.Till now we were trying everything locally on a cluster of
> 4 machines created by us, but now we are planning to move to AWS.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Tue, Dec 20, 2011 at 2:37 AM, Andrei Savu <sa...@gmail.com>
> wrote:
> > Hi,
> >
> > And welcome to the community! I am happy to see that you've been able to
> get
> > up and
> > running by following the instructions on the website.
> >
> > Hive is still work in
> > progress: https://issues.apache.org/jira/browse/WHIRR-343 and the
> > same is true for Pig.
> >
> > You can easily add any service you want. Check the following
> presentation:
> > http://www.oscon.com/oscon2011/public/schedule/detail/19214
> > (it explain both how Whirr works and how to write a service from scratch)
> >
> > If you want to take the lead on implementing a new service I can provide
> > as much assistance as needed on the way.
> >
> > Can you tell us a bit more about the end product you are building on
> Whirr?
> >
> > Cheers,
> >
> > -- Andrei Savu / andreisavu.ro
> >
> >
> > On Mon, Dec 19, 2011 at 9:42 PM, Mohammad Tariq <do...@gmail.com>
> wrote:
> >>
> >> Hello list,
> >>
> >> I am totally new to Whirr and just trying to get a proper feel of
> >> it..I was going through the wiki and with the help of documentation
> >> there I was able to create a cluster that has Hadoop and Hbase..Now I
> >> want to create a proper working cluster that has Hadoop, Hbase, Hive
> >> and Pig together.How can I do it?Also I would like to know, whether it
> >> is possible to add new tools(like Chukwa, Sqoop etc) to my existing
> >> Hadoop cluster using Whirr or not.If yes, then what should be the
> >> approach???Many thanks.
> >>
> >> Regards,
> >> Mohammad Tariq
> >
> >
>
Re: Creating a Hadoop cluster(Hbase+Hive+pig)
Posted by Mohammad Tariq <do...@gmail.com>.
Hi Andrei,
First of all I would like to thank you guys for the great work
you all have done.Actually, we are trying to set up a Hadoop cluster
on Amazon that includes Chukwa, Hbase, Hive and Pig.We'll use this
setup to collect data from remote servers that is coming in xml format
using Chukwa.And we'll dump this data to HDFS and Hbase.Then using Pig
we'll do some analytics and may be some indexing through Solr etc.So,
I was trying to figure out the options we have and I found Whirr most
appropriate.Till now we were trying everything locally on a cluster of
4 machines created by us, but now we are planning to move to AWS.
Regards,
Mohammad Tariq
On Tue, Dec 20, 2011 at 2:37 AM, Andrei Savu <sa...@gmail.com> wrote:
> Hi,
>
> And welcome to the community! I am happy to see that you've been able to get
> up and
> running by following the instructions on the website.
>
> Hive is still work in
> progress: https://issues.apache.org/jira/browse/WHIRR-343 and the
> same is true for Pig.
>
> You can easily add any service you want. Check the following presentation:
> http://www.oscon.com/oscon2011/public/schedule/detail/19214
> (it explain both how Whirr works and how to write a service from scratch)
>
> If you want to take the lead on implementing a new service I can provide
> as much assistance as needed on the way.
>
> Can you tell us a bit more about the end product you are building on Whirr?
>
> Cheers,
>
> -- Andrei Savu / andreisavu.ro
>
>
> On Mon, Dec 19, 2011 at 9:42 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>
>> Hello list,
>>
>> I am totally new to Whirr and just trying to get a proper feel of
>> it..I was going through the wiki and with the help of documentation
>> there I was able to create a cluster that has Hadoop and Hbase..Now I
>> want to create a proper working cluster that has Hadoop, Hbase, Hive
>> and Pig together.How can I do it?Also I would like to know, whether it
>> is possible to add new tools(like Chukwa, Sqoop etc) to my existing
>> Hadoop cluster using Whirr or not.If yes, then what should be the
>> approach???Many thanks.
>>
>> Regards,
>> Mohammad Tariq
>
>
Re: Creating a Hadoop cluster(Hbase+Hive+pig)
Posted by Andrei Savu <sa...@gmail.com>.
Hi,
And welcome to the community! I am happy to see that you've been able to
get up and
running by following the instructions on the website.
Hive is still work in progress:
https://issues.apache.org/jira/browse/WHIRR-343 and the
same is true for Pig.
You can easily add any service you want. Check the following presentation:
http://www.oscon.com/oscon2011/public/schedule/detail/19214
(it explain both how Whirr works and how to write a service from scratch)
If you want to take the lead on implementing a new service I can provide
as much assistance as needed on the way.
Can you tell us a bit more about the end product you are building on Whirr?
Cheers,
-- Andrei Savu / andreisavu.ro
On Mon, Dec 19, 2011 at 9:42 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello list,
>
> I am totally new to Whirr and just trying to get a proper feel of
> it..I was going through the wiki and with the help of documentation
> there I was able to create a cluster that has Hadoop and Hbase..Now I
> want to create a proper working cluster that has Hadoop, Hbase, Hive
> and Pig together.How can I do it?Also I would like to know, whether it
> is possible to add new tools(like Chukwa, Sqoop etc) to my existing
> Hadoop cluster using Whirr or not.If yes, then what should be the
> approach???Many thanks.
>
> Regards,
> Mohammad Tariq
>