Re: polishing off bigtop : SSH Provisioning to the cloud

[Konstantin Boudnik <co...@apache.org>]

Count me in.

Thanks!
  Cos

On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote:
> We set up a goto meeting for 10:00AM PST on Thursday March 5th
> 
> *Bigtop - DevOps Toolkit (DTK) Demo*
> Thu, Mar 5, 10:00 AM Pacific Standard Time
> 
> 
>    - Please join my meeting from your computer, tablet or smartphone.
>    https://global.gotomeeting.com/join/655931173
>    - You can also dial in using your phone.
>     United States (Long distance): +1 (571) 317-3122
>     *Access Code:* 655-931-173
>    More phone numbers:
>    https://global.gotomeeting.com/655931173/numbersdisplay.html
> 
> If interested, but this day or time does not work, please send a note and
> we can set up additional day/time(s) too.
> 
> We will have a few high level slides and then have a demo that shows how we
> currently use Bigtop and how this can be naturally extended .to provide a
> "one click" solution to deploy a rich variety of bigtop stacks with
> different mixtures of services and different topologies that plugs on top
> of the existing bigtop deployment work
> 
> As far as a screencast, we wanted  our first one to be a short focused
> screencast; so that will be a follow up that we will put together shortly
> - Rich
> 
> 
> 
> 
> ----- Original Message -----
> From:
>  dev@bigtop.apache.org
> 
> To:
> <de...@bigtop.apache.org>
> Cc:
> <us...@bigtop.apache.org>
> Sent:
> Sat, 28 Feb 2015 01:31:10 +0000
> Subject:
> Re: polishing off bigtop : SSH Provisioning to the cloud
> 
> 
> Either of the days is open for me: would love to have a demo. Perhaps it can
> be recorded to be used as a screencast later on? ;)
> 
> It'd be great to have as many people as possible on that, so we can quickly
> converge on the directions here. Adding user@ list as well...
> 
> Cos
> 
> On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> > Thanks Jay,
> > A screencast is something we have had on our todo list; we can easily
> > prioritize a bigtop example as the first of a few examples we were
> planning
> > to do.
> >
> > As far as the demo, some dates we had in mind were Wednesday or Thursday
> > next week.
> > I guess the easiest way to see who is interested is a reply/+1
> > with Wednesday or Thursday; we then can find exact times.
> > We are happy to schedule multiple demos at a few different days/times.
> > -Rich
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> > "dev@bigtop.apache.org" <de...@bigtop.apache.org>
> > Cc:
> >
> > Sent:
> > Wed, 25 Feb 2015 12:32:59 -0500
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud
> >
> >
> > Hi richard, also maybe even a youtube video of Devops TK deploying BigTop
> > would be awesome !
> >
> > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com>
> wrote:
> >
> > > I think might have a solution (dtk.io) that we have worked on that can
> > > help
> > > solve these problems. We have been working the last four years on the
> > > problem of deploying complex clusters. It provides a simple DSL for
> > > describing a rich set of topologies (eg simple master slave topologies,
> HA
> > > variants, security variants, monitoring and centralized logging
> > > extensions). It provides one click deployment from the selected cluster
> > > topology. It also provides a simple workflow DSL to enable finer grain
> > > control on order of execution.
> > >
> > > We are in the process now of refining a plan to open source it
> > >
> > > Our system was built to work on top of user's existing configuration
> > > assets. The initial focus has been on Puppet, but we are expanding now
> for
> > > docker support and using arbitrary scripts
> > >
> > > For last two years we have been using some of the Bigtop puppet modules
> > for
> > > our big data deployments as well as in concert with modules such as
> kafka,
> > > storm, accumulo , opentsdb ,.. We are in process now of converting to
> use
> > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
> > > leverage the groovy and gradle test work so one can bring up cluster
> stage
> > > by stage with smoke tests after every stage
> > >
> > > Think it would be straight forward so this can also plug on top of
> Bigtop
> > > vagrant work; we have a plugable iaas architecture; we currently support
> > > ec2 and "managed servers" (we have focused here on bare metal); so
> vagrant
> > > would extend reach to virtual box and its other iaas providers.
> > >
> > > I think there is potential for great synergy; best way to show would be
> > > through
> > > goto meeting demo(s) and answering any detailed questions. We can do
> this
> > > next week or the following at one or a few sessions. Will shortly send
> > > proposed dates/times
> > > - Rich
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From:
> > > dev@bigtop.apache.org
> > >
> > > To:
> > > <de...@bigtop.apache.org>
> > > Cc:
> > >
> > > Sent:
> > > Wed, 25 Feb 2015 17:24:41 +0800
> > > Subject:
> > > Re: polishing off bigtop : SSH Provisioning to the cloud.
> > >
> > >
> > > Cluster deployment and management are important things to make our work
> > > finally adopted by users. From my point of view I think there are two
> > > things we can improve further:
> > > * user friendly cluster auto deployment
> > > (we have puppet recipes, but users still need to manually ship puppet
> > > code and configurations)
> > > * topology layout
> > > (for example, specify zookeeper to be deployed on node 3~5)
> > >
> > > BIGTOP-1702 has a great potential to make cluster deployment more user
> > > friendly.
> > > That is users don't need to deal with those trivial things like ship
> > bigtop
> > > puppet code and hiera configurations or install those prerequisites all
> > > over the cluster.
> > >
> > > To answer cos's question in BIGTOP-1702:
> > >
> > > although I am not sure what you mean by
> > > link those managed servers for further provisioning
> > >
> > > According to the readme of vagrant-managed-servers, vagrant up and
> destroy
> > > are re-interpreted as "linking//unlinking". After those servers are
> > > linked, theoretically, we can directly leverage our Vagrantfile
> > > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
> > > So, this definitely will take advantage of existing puppet recipes.
> > >
> > > And here's my thought on following questions:
> > >
> > > - how that topology will be specified?
> > > - how it is going to be compatible with what we have right now? Not
> like I
> > > try to preserve what we have right now, as there's not much to preserve.
> > > We don't really have a way to desribe a cluster's topology at the
> > > moment anyway.
> > > - in case we need to describe more complex topology - how would be done?
> > >
> > > I don't think BIGTOP-1702 support topology settings. Regarding to the
> > > feature, I think we should support topology specification in our puppet
> > > recipes. We currently do not support this yet, but I believe Michael is
> > > heading toward this way.
> > >
> > >
> > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > I think it is a great idea!
> > > >
> > > > And, indeed, it's just a very small step - make it user-friendly ;)
> > > >
> > > > I believe what you're proposing below is a much better way to achieve
> > > what
> > > > I've been trying by wrapping Puppet and content distribution into
> Gradle
> > > > (you
> > > > referred to it 'our own deployer' below). That was a bad idea, in
> > > > retrospect.
> > > > And for sure - we need the functionality of this kind: we need
> something
> > > > that
> > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly
> > without
> > > > additional troubles!
> > > >
> > > > I am also a big fan of getting rid of lengthy and at times cryptic
> pages
> > > > explaining the installation process.
> > > >
> > > > Now, the solution you have in mind will be helping with a typical
> > cluster
> > > > topology like we are usually setting up with head_node, and other flat
> > > > things
> > > > of this nature, right? In other words - a simple, "traditional"
> cluster
> > > > layout? If so, may I ask a few questions:
> > > > - how that topology will be specified?
> > > > - how it is going to be compatible with what we have right now? Not
> like
> > > I
> > > > try to preserve what we have right now, as there's not much to
> preserve.
> > > > We don't really have a way to desribe a cluster's topology at the
> > > > moment anyway.
> > > > - in case we need to describe more complex topology - how would be
> done?
> > > >
> > > > I also presume, that the proposed solution will take advantage of
> > > existing
> > > > Puppet recipes, right?
> > > >
> > > > Thanks in advance for your answers!
> > > > Cos
> > > >
> > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > > Hi folks.
> > > > >
> > > > > This last year we've made a lot of progress,
> > > > > - we can now easily build bigtop,
> > > > > - and we can test a fresh build in output/ it in vagrant boxes
> locally
> > > > >
> > > > > Whats next?
> > > > >
> > > > > In my mind, there is only one last step, to make bigtop a consumer
> > > facing
> > > > > product: We need to make it so you can simply provision a cluster in
> > > the
> > > > > cloud.
> > > > > As you may recall, David (student at RPI, intern at analytics
> company
> > > > last
> > > > > summer), asked how to do this, and spent about three weeks on it.
> > > > Recently
> > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
> University,
> > > and
> > > > > former Yahoo researchers, also asked me the same thing.
> > > > >
> > > > > Alternative?
> > > > >
> > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so
> > > on,
> > > > > which mostly are an untested, human readable version of our vagrant
> > > > > recipes. yikes !
> > > > >
> > > > > How to implement a ssh cloud bigtop installation ?
> > > > >
> > > > > We could roll our own deployer, but in doing so, we would have to
> make
> > > > > semantics for:
> > > > >
> > > > > - possible need for machiene reboots
> > > > > - syncing local folders to remote folders
> > > > > - installing (and reinstalling)
> > > > >
> > > > > luckily, our buddy Vagrant already does this for us (respectively ,
> > > with
> > > > > the "reload","synced.folder", and "up" options).
> > > > >
> > > > > Additionally, *** I THINK *** we can literally use the exact same
> > > vagrant
> > > > > recipes which we are already using to test --- so we will have a
> great
> > > > user
> > > > > experience, and really easy to reproduce bugs and test deployments.
> > > > >
> > > > >
> > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > > > suggestions
> > > > > on the implementation or feature? I think it will be easy, but also,
> > it
> > > > > will be one of the most powerful things we add to bigtop, basically
> > > > > allowing users to easily use the system... maybe even too easy :)
> > > > >
> > > > > --
> > > > > jay vyas
> > > >
> > >
> >
> >
> >
> > --
> > jay vyas