Re: polishing off bigtop : SSH Provisioning to the cloud

[Richard Pelavin <ri...@reactor8.com>]

I think might have a solution (dtk.io) that we have worked on that can help
solve these problems. We have been working the last four years on the
problem of deploying complex clusters. It provides a simple DSL for
describing a rich set of topologies (eg simple master slave topologies, HA
variants, security variants, monitoring and centralized logging
extensions). It provides one click deployment from the selected cluster
topology. It also provides a simple workflow DSL to enable finer grain
control on order of execution.

We are in the process now of refining a plan to open source it

Our system was built to work on top of user's existing configuration
assets. The initial focus has been on Puppet, but we are expanding now for
docker support and using arbitrary scripts

For last two years we have been using some of the Bigtop puppet modules for
our big data deployments as well as in concert with modules such as kafka,
storm, accumulo , opentsdb ,.. We are in process now of converting to use
the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
leverage the groovy and gradle test work so one can bring up cluster stage
by stage with smoke tests after every stage

Think it would be straight forward so this can also plug on top of Bigtop
vagrant work; we have a plugable iaas architecture; we currently support
ec2 and "managed servers" (we have focused here on bare metal);  so vagrant
would extend reach to virtual box and its other iaas providers.

I think there is potential for great synergy; best way to show would be
through
goto meeting demo(s) and answering any detailed questions. We can do this
next week or the following at one or a few sessions. Will shortly send
proposed dates/times
- Rich



----- Original Message -----
From:
 dev@bigtop.apache.org

To:
<de...@bigtop.apache.org>
Cc:

Sent:
Wed, 25 Feb 2015 17:24:41 +0800
Subject:
Re: polishing off bigtop : SSH Provisioning to the cloud.


Cluster deployment and management are important things to make our work
finally adopted by users. From my point of view I think there are two
things we can improve further:
* user friendly cluster auto deployment
(we have puppet recipes, but users still need to manually ship puppet
code and configurations)
* topology layout
(for example, specify zookeeper to be deployed on node 3~5)

BIGTOP-1702 has a great potential to make cluster deployment more user
friendly.
That is users don't need to deal with those trivial things like ship bigtop
puppet code and hiera configurations or install those prerequisites all
over the cluster.

To answer cos's question in BIGTOP-1702:

although I am not sure what you mean by
link those managed servers for further provisioning

According to the readme of vagrant-managed-servers, vagrant up and destroy
are re-interpreted as "linking//unlinking". After those servers are
linked, theoretically, we can directly leverage our Vagrantfile
in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
So, this definitely will take advantage of existing puppet recipes.

And here's my thought on following questions:

- how that topology will be specified?
- how it is going to be compatible with what we have right now? Not like I
try to preserve what we have right now, as there's not much to preserve.
We don't really have a way to desribe a cluster's topology at the
moment anyway.
- in case we need to describe more complex topology - how would be done?

I don't think BIGTOP-1702 support topology settings. Regarding to the
feature, I think we should support topology specification in our puppet
recipes. We currently do not support this yet, but I believe Michael is
heading toward this way.


2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:

> I think it is a great idea!
>
> And, indeed, it's just a very small step - make it user-friendly ;)
>
> I believe what you're proposing below is a much better way to achieve what
> I've been trying by wrapping Puppet and content distribution into Gradle
> (you
> referred to it 'our own deployer' below). That was a bad idea, in
> retrospect.
> And for sure - we need the functionality of this kind: we need something
> that
> let anyone to deploy a cluster to a pre-provisioned nodes quickly without
> additional troubles!
>
> I am also a big fan of getting rid of lengthy and at times cryptic pages
> explaining the installation process.
>
> Now, the solution you have in mind will be helping with a typical cluster
> topology like we are usually setting up with head_node, and other flat
> things
> of this nature, right? In other words - a simple, "traditional" cluster
> layout? If so, may I ask a few questions:
> - how that topology will be specified?
> - how it is going to be compatible with what we have right now? Not like I
> try to preserve what we have right now, as there's not much to preserve.
> We don't really have a way to desribe a cluster's topology at the
> moment anyway.
> - in case we need to describe more complex topology - how would be done?
>
> I also presume, that the proposed solution will take advantage of existing
> Puppet recipes, right?
>
> Thanks in advance for your answers!
> Cos
>
> On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > Hi folks.
> >
> > This last year we've made a lot of progress,
> > - we can now easily build bigtop,
> > - and we can test a fresh build in output/ it in vagrant boxes locally
> >
> > Whats next?
> >
> > In my mind, there is only one last step, to make bigtop a consumer
facing
> > product: We need to make it so you can simply provision a cluster in the
> > cloud.
> > As you may recall, David (student at RPI, intern at analytics company
> last
> > summer), asked how to do this, and spent about three weeks on it.
> Recently
> > Ata Turk, working on the Massachusets Open Cloud @ Boston University,
and
> > former Yahoo researchers, also asked me the same thing.
> >
> > Alternative?
> >
> > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so on,
> > which mostly are an untested, human readable version of our vagrant
> > recipes. yikes !
> >
> > How to implement a ssh cloud bigtop installation ?
> >
> > We could roll our own deployer, but in doing so, we would have to make
> > semantics for:
> >
> > - possible need for machiene reboots
> > - syncing local folders to remote folders
> > - installing (and reinstalling)
> >
> > luckily, our buddy Vagrant already does this for us (respectively , with
> > the "reload","synced.folder", and "up" options).
> >
> > Additionally, *** I THINK *** we can literally use the exact same
vagrant
> > recipes which we are already using to test --- so we will have a great
> user
> > experience, and really easy to reproduce bugs and test deployments.
> >
> >
> > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> suggestions
> > on the implementation or feature? I think it will be easy, but also, it
> > will be one of the most powerful things we add to bigtop, basically
> > allowing users to easily use the system... maybe even too easy :)
> >
> > --
> > jay vyas
>

Re: polishing off bigtop : SSH Provisioning to the cloud

[jay vyas <ja...@gmail.com>]

Hi richard, also maybe even a youtube video of Devops TK deploying BigTop
would be awesome !

On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com> wrote:

> I think might have a solution (dtk.io) that we have worked on that can
> help
> solve these problems. We have been working the last four years on the
> problem of deploying complex clusters. It provides a simple DSL for
> describing a rich set of topologies (eg simple master slave topologies, HA
> variants, security variants, monitoring and centralized logging
> extensions). It provides one click deployment from the selected cluster
> topology. It also provides a simple workflow DSL to enable finer grain
> control on order of execution.
>
> We are in the process now of refining a plan to open source it
>
> Our system was built to work on top of user's existing configuration
> assets. The initial focus has been on Puppet, but we are expanding now for
> docker support and using arbitrary scripts
>
> For last two years we have been using some of the Bigtop puppet modules for
> our big data deployments as well as in concert with modules such as kafka,
> storm, accumulo , opentsdb ,.. We are in process now of converting to use
> the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
> leverage the groovy and gradle test work so one can bring up cluster stage
> by stage with smoke tests after every stage
>
> Think it would be straight forward so this can also plug on top of Bigtop
> vagrant work; we have a plugable iaas architecture; we currently support
> ec2 and "managed servers" (we have focused here on bare metal);  so vagrant
> would extend reach to virtual box and its other iaas providers.
>
> I think there is potential for great synergy; best way to show would be
> through
> goto meeting demo(s) and answering any detailed questions. We can do this
> next week or the following at one or a few sessions. Will shortly send
> proposed dates/times
> - Rich
>
>
>
> ----- Original Message -----
> From:
>  dev@bigtop.apache.org
>
> To:
> <de...@bigtop.apache.org>
> Cc:
>
> Sent:
> Wed, 25 Feb 2015 17:24:41 +0800
> Subject:
> Re: polishing off bigtop : SSH Provisioning to the cloud.
>
>
> Cluster deployment and management are important things to make our work
> finally adopted by users. From my point of view I think there are two
> things we can improve further:
> * user friendly cluster auto deployment
> (we have puppet recipes, but users still need to manually ship puppet
> code and configurations)
> * topology layout
> (for example, specify zookeeper to be deployed on node 3~5)
>
> BIGTOP-1702 has a great potential to make cluster deployment more user
> friendly.
> That is users don't need to deal with those trivial things like ship bigtop
> puppet code and hiera configurations or install those prerequisites all
> over the cluster.
>
> To answer cos's question in BIGTOP-1702:
>
> although I am not sure what you mean by
> link those managed servers for further provisioning
>
> According to the readme of vagrant-managed-servers, vagrant up and destroy
> are re-interpreted as "linking//unlinking". After those servers are
> linked, theoretically, we can directly leverage our Vagrantfile
> in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
> So, this definitely will take advantage of existing puppet recipes.
>
> And here's my thought on following questions:
>
> - how that topology will be specified?
> - how it is going to be compatible with what we have right now? Not like I
> try to preserve what we have right now, as there's not much to preserve.
> We don't really have a way to desribe a cluster's topology at the
> moment anyway.
> - in case we need to describe more complex topology - how would be done?
>
> I don't think BIGTOP-1702 support topology settings. Regarding to the
> feature, I think we should support topology specification in our puppet
> recipes. We currently do not support this yet, but I believe Michael is
> heading toward this way.
>
>
> 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
>
> > I think it is a great idea!
> >
> > And, indeed, it's just a very small step - make it user-friendly ;)
> >
> > I believe what you're proposing below is a much better way to achieve
> what
> > I've been trying by wrapping Puppet and content distribution into Gradle
> > (you
> > referred to it 'our own deployer' below). That was a bad idea, in
> > retrospect.
> > And for sure - we need the functionality of this kind: we need something
> > that
> > let anyone to deploy a cluster to a pre-provisioned nodes quickly without
> > additional troubles!
> >
> > I am also a big fan of getting rid of lengthy and at times cryptic pages
> > explaining the installation process.
> >
> > Now, the solution you have in mind will be helping with a typical cluster
> > topology like we are usually setting up with head_node, and other flat
> > things
> > of this nature, right? In other words - a simple, "traditional" cluster
> > layout? If so, may I ask a few questions:
> > - how that topology will be specified?
> > - how it is going to be compatible with what we have right now? Not like
> I
> > try to preserve what we have right now, as there's not much to preserve.
> > We don't really have a way to desribe a cluster's topology at the
> > moment anyway.
> > - in case we need to describe more complex topology - how would be done?
> >
> > I also presume, that the proposed solution will take advantage of
> existing
> > Puppet recipes, right?
> >
> > Thanks in advance for your answers!
> > Cos
> >
> > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > Hi folks.
> > >
> > > This last year we've made a lot of progress,
> > > - we can now easily build bigtop,
> > > - and we can test a fresh build in output/ it in vagrant boxes locally
> > >
> > > Whats next?
> > >
> > > In my mind, there is only one last step, to make bigtop a consumer
> facing
> > > product: We need to make it so you can simply provision a cluster in
> the
> > > cloud.
> > > As you may recall, David (student at RPI, intern at analytics company
> > last
> > > summer), asked how to do this, and spent about three weeks on it.
> > Recently
> > > Ata Turk, working on the Massachusets Open Cloud @ Boston University,
> and
> > > former Yahoo researchers, also asked me the same thing.
> > >
> > > Alternative?
> > >
> > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so
> on,
> > > which mostly are an untested, human readable version of our vagrant
> > > recipes. yikes !
> > >
> > > How to implement a ssh cloud bigtop installation ?
> > >
> > > We could roll our own deployer, but in doing so, we would have to make
> > > semantics for:
> > >
> > > - possible need for machiene reboots
> > > - syncing local folders to remote folders
> > > - installing (and reinstalling)
> > >
> > > luckily, our buddy Vagrant already does this for us (respectively ,
> with
> > > the "reload","synced.folder", and "up" options).
> > >
> > > Additionally, *** I THINK *** we can literally use the exact same
> vagrant
> > > recipes which we are already using to test --- so we will have a great
> > user
> > > experience, and really easy to reproduce bugs and test deployments.
> > >
> > >
> > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > suggestions
> > > on the implementation or feature? I think it will be easy, but also, it
> > > will be one of the most powerful things we add to bigtop, basically
> > > allowing users to easily use the system... maybe even too easy :)
> > >
> > > --
> > > jay vyas
> >
>



-- 
jay vyas