You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@bigtop.apache.org by Demai Ni <ni...@gmail.com> on 2015/06/06 00:30:39 UTC
newbie question of BigTop
hi, folks,
I am looking at BigTop as noncommercial hadoop distribution package, as an
alternative of CDH, HDP. And this is my first time to try out bigTop.
My expectation was:
1) setup my cluster with passwordless ssh
2) download the necessary bigtop components on one of the nodes.
3) run an installation/configure tool from the node on 2)
4) BigTop will (through puppet) install and configure the whole cluster.
the point is that no need of a lot of manual work on every node.
But I am following instruction for 0.8.0 and 0.7.0, like
https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet
there are quite a few steps, (almost all the steps in above link) are
required to be performed on each node?
so it looks like the human work is significant higher between small(3 node)
and large(100+) cluster. Is that right?
Thanks.
Demai
Re: newbie question of BigTop
Posted by jay vyas <ja...@gmail.com>.
also, not sure if i put this link up for you yet, but here :
https://www.youtube.com/watch?v=4GfcKEjO6e8&feature=youtu.be .
its a bigtop internals explanation which shows some info about how the
vagrant and puppet stuff works.
you will be ready to hack bigtop w/ the best of em after watching that :)
On Mon, Jun 8, 2015 at 8:02 PM, Demai Ni <ni...@gmail.com> wrote:
> Evans,
>
> thank you very much for the detail instructions.
>
> I understood that the cluster need to be passwordless ssh or someway to
> provide userid/password of all nodes. So that actions can be propogated to
> other nodes.
>
> I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
> would do such job for end-users. But the document didn't mention such
> tool(or I missed them?), and I got the impression that that steps 1)~3) are
> required by manual effort.
>
> Maybe those tools are common-sense for an experienced cluster administer,
> so no document is necessary.:-) Anyway, now that with your help, I will try
> them out. Thanks
>
> Demai
>
> On Sat, Jun 6, 2015 at 12:03 PM, Evans Ye <ev...@apache.org> wrote:
>
>> The missing note in the document is that the cluster admin need to setup
>> some general environment settings such as
>> the password-less ssh. Definitely it would be great if we can provide some
>> suggestions or ways to do that, but I think somehow you still need to setup
>> that during machine provisioning(installing OS, setting up hostname,
>> ip, password, ssh keys), right?
>>
>> If you have bunch of machines ready for the deployment, here's the step
>> to setup a bigtop cluster:
>>
>> 1) install puppet on each node
>> 2) install puppet modules on each node
>>
>> puppet apply --modulepath= -e "include bigtop_toolchain::puppet-modules"
>>
>> 2) propagate puppet recipes and configurations
>> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet>on
>> each node
>> 3) run puppet apply on each node
>> 4) done!
>>
>> The puppet and module installation can be done through cluster admin tool
>> such as pdsh, which allows you to run same command across the whole
>> cluster in parallel mode. The puppet recipes or bigtop repo can be put
>> in a share storage such as NFS so that it is available for all nodes. After
>> all of this are set, you can then run puppet apply on each node through
>> pdsh as well. This should give you an up and running bigtop cluster.
>>
>> If you're not quite sure how to do that, you can also refer to bigtop's vagrant
>> provisioner
>> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet-vm>,
>> which demonstrate the whole process to setup a bigtop cluster by a click of
>> button.
>>
>> Of course, you can ask questions in mailing list when you encounter any
>> issue. We're happy to help. :)
>>
>> 2015-06-06 6:30 GMT+08:00 Demai Ni <ni...@gmail.com>:
>>
>>> hi, folks,
>>>
>>> I am looking at BigTop as noncommercial hadoop distribution package, as
>>> an alternative of CDH, HDP. And this is my first time to try out bigTop.
>>>
>>> My expectation was:
>>> 1) setup my cluster with passwordless ssh
>>> 2) download the necessary bigtop components on one of the nodes.
>>> 3) run an installation/configure tool from the node on 2)
>>> 4) BigTop will (through puppet) install and configure the whole cluster.
>>>
>>> the point is that no need of a lot of manual work on every node.
>>>
>>> But I am following instruction for 0.8.0 and 0.7.0, like
>>>
>>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet
>>>
>>> there are quite a few steps, (almost all the steps in above link) are
>>> required to be performed on each node?
>>>
>>> so it looks like the human work is significant higher between small(3
>>> node) and large(100+) cluster. Is that right?
>>>
>>> Thanks.
>>>
>>> Demai
>>>
>>>
>>
>>
>
--
jay vyas
Re: newbie question of BigTop
Posted by Jay Vyas <ja...@gmail.com>.
This is cool Martin! Thanks.
As you know we are heavily tied to puppet.
But since this doesn't duplicate functionality but rather Provides ansible wrappers it might be useful to some folks.
> On Jul 2, 2015, at 1:21 PM, Martin Bukatovic <mb...@redhat.com> wrote:
>
> Dear bigtop users,
>
>> On 06/10/2015 06:50 PM, Martin Bukatovic wrote:
>>> On 06/09/2015 02:02 AM, Demai Ni wrote:
>>> I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
>>> would do such job for end-users. But the document didn't mention such
>>> tool(or I missed them?), and I got the impression that that steps 1)~3)
>>> are required by manual effort.
>>
>> For this I use ansible playbook which suits my needs well as I have
>> small testing 3 machine cluster. But this is hardly something which
>> suits the needs of the bigtop project I guess.
>
> I have published initial version of ansible playbook which sets
> up and runs bigtop masterless puppet recipe as described in my
> previous comment. It works with current master only (upcoming
> 1.0.0 release).
>
> https://github.com/mbukatov/bigtop-contrib/tree/master/playbook
>
> This doesn't make much sense to be included in bigtop, but
> since I have seen some interest in this area here, I'm
> sharing my scripts with you.
>
> There actually is ansible module for puppet, see:
>
> http://docs.ansible.com/puppet_module.html
>
> but I'm not using it yet.
>
> --
> Martin Bukatovic
> RHS/Hadoop QE Team
Re: newbie question of BigTop
Posted by 김영우 <wa...@gmail.com>.
Hi Martin,
Looks cool!
Actually I've done some work on Ansible and Bigtop, but unfortunately dont
have cycles for that at this moment.
https://issues.apache.org/jira/browse/BIGTOP-1584
I believe your scripts are very useful for Ansible users. Eventually it
would be nice to have a native Ansible stuffs for Bigtop and Ansible users.
Thanks,
Youngwoo
On Fri, Jul 3, 2015 at 2:21 AM, Martin Bukatovic <mb...@redhat.com>
wrote:
> Dear bigtop users,
>
> On 06/10/2015 06:50 PM, Martin Bukatovic wrote:
> > On 06/09/2015 02:02 AM, Demai Ni wrote:
> >> I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
> >> would do such job for end-users. But the document didn't mention such
> >> tool(or I missed them?), and I got the impression that that steps 1)~3)
> >> are required by manual effort.
> >
> > For this I use ansible playbook which suits my needs well as I have
> > small testing 3 machine cluster. But this is hardly something which
> > suits the needs of the bigtop project I guess.
>
> I have published initial version of ansible playbook which sets
> up and runs bigtop masterless puppet recipe as described in my
> previous comment. It works with current master only (upcoming
> 1.0.0 release).
>
> https://github.com/mbukatov/bigtop-contrib/tree/master/playbook
>
> This doesn't make much sense to be included in bigtop, but
> since I have seen some interest in this area here, I'm
> sharing my scripts with you.
>
> There actually is ansible module for puppet, see:
>
> http://docs.ansible.com/puppet_module.html
>
> but I'm not using it yet.
>
> --
> Martin Bukatovic
> RHS/Hadoop QE Team
>
Re: newbie question of BigTop
Posted by Martin Bukatovic <mb...@redhat.com>.
Dear bigtop users,
On 06/10/2015 06:50 PM, Martin Bukatovic wrote:
> On 06/09/2015 02:02 AM, Demai Ni wrote:
>> I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
>> would do such job for end-users. But the document didn't mention such
>> tool(or I missed them?), and I got the impression that that steps 1)~3)
>> are required by manual effort.
>
> For this I use ansible playbook which suits my needs well as I have
> small testing 3 machine cluster. But this is hardly something which
> suits the needs of the bigtop project I guess.
I have published initial version of ansible playbook which sets
up and runs bigtop masterless puppet recipe as described in my
previous comment. It works with current master only (upcoming
1.0.0 release).
https://github.com/mbukatov/bigtop-contrib/tree/master/playbook
This doesn't make much sense to be included in bigtop, but
since I have seen some interest in this area here, I'm
sharing my scripts with you.
There actually is ansible module for puppet, see:
http://docs.ansible.com/puppet_module.html
but I'm not using it yet.
--
Martin Bukatovic
RHS/Hadoop QE Team
Re: newbie question of BigTop
Posted by Martin Bukatovic <mb...@redhat.com>.
On 06/09/2015 02:02 AM, Demai Ni wrote:
> I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
> would do such job for end-users. But the document didn't mention such
> tool(or I missed them?), and I got the impression that that steps 1)~3)
> are required by manual effort.
For this I use ansible playbook which suits my needs well as I have
small testing 3 machine cluster. But this is hardly something which
suits the needs of the bigtop project I guess.
--
Martin Bukatovic
RHS/Hadoop QE Team
Re: newbie question of BigTop
Posted by Demai Ni <ni...@gmail.com>.
Evans,
thank you very much for the detail instructions.
I understood that the cluster need to be passwordless ssh or someway to
provide userid/password of all nodes. So that actions can be propogated to
other nodes.
I was expecting some tool (pdsh, vagrant provisioner) as you mentioned,
would do such job for end-users. But the document didn't mention such
tool(or I missed them?), and I got the impression that that steps 1)~3) are
required by manual effort.
Maybe those tools are common-sense for an experienced cluster administer,
so no document is necessary.:-) Anyway, now that with your help, I will try
them out. Thanks
Demai
On Sat, Jun 6, 2015 at 12:03 PM, Evans Ye <ev...@apache.org> wrote:
> The missing note in the document is that the cluster admin need to setup
> some general environment settings such as
> the password-less ssh. Definitely it would be great if we can provide some
> suggestions or ways to do that, but I think somehow you still need to setup
> that during machine provisioning(installing OS, setting up hostname,
> ip, password, ssh keys), right?
>
> If you have bunch of machines ready for the deployment, here's the step
> to setup a bigtop cluster:
>
> 1) install puppet on each node
> 2) install puppet modules on each node
>
> puppet apply --modulepath= -e "include bigtop_toolchain::puppet-modules"
>
> 2) propagate puppet recipes and configurations
> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet>on
> each node
> 3) run puppet apply on each node
> 4) done!
>
> The puppet and module installation can be done through cluster admin tool
> such as pdsh, which allows you to run same command across the whole
> cluster in parallel mode. The puppet recipes or bigtop repo can be put in
> a share storage such as NFS so that it is available for all nodes. After
> all of this are set, you can then run puppet apply on each node through
> pdsh as well. This should give you an up and running bigtop cluster.
>
> If you're not quite sure how to do that, you can also refer to bigtop's vagrant
> provisioner
> <https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet-vm>,
> which demonstrate the whole process to setup a bigtop cluster by a click of
> button.
>
> Of course, you can ask questions in mailing list when you encounter any
> issue. We're happy to help. :)
>
> 2015-06-06 6:30 GMT+08:00 Demai Ni <ni...@gmail.com>:
>
>> hi, folks,
>>
>> I am looking at BigTop as noncommercial hadoop distribution package, as
>> an alternative of CDH, HDP. And this is my first time to try out bigTop.
>>
>> My expectation was:
>> 1) setup my cluster with passwordless ssh
>> 2) download the necessary bigtop components on one of the nodes.
>> 3) run an installation/configure tool from the node on 2)
>> 4) BigTop will (through puppet) install and configure the whole cluster.
>>
>> the point is that no need of a lot of manual work on every node.
>>
>> But I am following instruction for 0.8.0 and 0.7.0, like
>>
>> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet
>>
>> there are quite a few steps, (almost all the steps in above link) are
>> required to be performed on each node?
>>
>> so it looks like the human work is significant higher between small(3
>> node) and large(100+) cluster. Is that right?
>>
>> Thanks.
>>
>> Demai
>>
>>
>
>
Re: newbie question of BigTop
Posted by Evans Ye <ev...@apache.org>.
The missing note in the document is that the cluster admin need to setup
some general environment settings such as
the password-less ssh. Definitely it would be great if we can provide some
suggestions or ways to do that, but I think somehow you still need to setup
that during machine provisioning(installing OS, setting up hostname,
ip, password, ssh keys), right?
If you have bunch of machines ready for the deployment, here's the step to
setup a bigtop cluster:
1) install puppet on each node
2) install puppet modules on each node
puppet apply --modulepath= -e "include bigtop_toolchain::puppet-modules"
2) propagate puppet recipes and configurations
<https://github.com/apache/bigtop/tree/master/bigtop-deploy/puppet>on each
node
3) run puppet apply on each node
4) done!
The puppet and module installation can be done through cluster admin tool
such as pdsh, which allows you to run same command across the whole cluster
in parallel mode. The puppet recipes or bigtop repo can be put in a share
storage such as NFS so that it is available for all nodes. After all of
this are set, you can then run puppet apply on each node through pdsh as
well. This should give you an up and running bigtop cluster.
If you're not quite sure how to do that, you can also refer to bigtop's vagrant
provisioner
<https://github.com/apache/bigtop/tree/master/bigtop-deploy/vm/vagrant-puppet-vm>,
which demonstrate the whole process to setup a bigtop cluster by a click of
button.
Of course, you can ask questions in mailing list when you encounter any
issue. We're happy to help. :)
2015-06-06 6:30 GMT+08:00 Demai Ni <ni...@gmail.com>:
> hi, folks,
>
> I am looking at BigTop as noncommercial hadoop distribution package, as an
> alternative of CDH, HDP. And this is my first time to try out bigTop.
>
> My expectation was:
> 1) setup my cluster with passwordless ssh
> 2) download the necessary bigtop components on one of the nodes.
> 3) run an installation/configure tool from the node on 2)
> 4) BigTop will (through puppet) install and configure the whole cluster.
>
> the point is that no need of a lot of manual work on every node.
>
> But I am following instruction for 0.8.0 and 0.7.0, like
>
> https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+BigTop+0.7.0+hadoop+on+CentOS+with+puppet
>
> there are quite a few steps, (almost all the steps in above link) are
> required to be performed on each node?
>
> so it looks like the human work is significant higher between small(3
> node) and large(100+) cluster. Is that right?
>
> Thanks.
>
> Demai
>
>