You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Richard Pelavin <ri...@reactor8.com> on 2015/03/02 21:22:24 UTC
Re: polishing off bigtop : SSH Provisioning to the cloud
We set up a goto meeting for 10:00AM PST on Thursday March 5th
*Bigtop - DevOps Toolkit (DTK) Demo*
Thu, Mar 5, 10:00 AM Pacific Standard Time
- Please join my meeting from your computer, tablet or smartphone.
https://global.gotomeeting.com/join/655931173
- You can also dial in using your phone.
United States (Long distance): +1 (571) 317-3122
*Access Code:* 655-931-173
More phone numbers:
https://global.gotomeeting.com/655931173/numbersdisplay.html
If interested, but this day or time does not work, please send a note and
we can set up additional day/time(s) too.
We will have a few high level slides and then have a demo that shows how we
currently use Bigtop and how this can be naturally extended .to provide a
"one click" solution to deploy a rich variety of bigtop stacks with
different mixtures of services and different topologies that plugs on top
of the existing bigtop deployment work
As far as a screencast, we wanted our first one to be a short focused
screencast; so that will be a follow up that we will put together shortly
- Rich
----- Original Message -----
From:
dev@bigtop.apache.org
To:
<de...@bigtop.apache.org>
Cc:
<us...@bigtop.apache.org>
Sent:
Sat, 28 Feb 2015 01:31:10 +0000
Subject:
Re: polishing off bigtop : SSH Provisioning to the cloud
Either of the days is open for me: would love to have a demo. Perhaps it can
be recorded to be used as a screencast later on? ;)
It'd be great to have as many people as possible on that, so we can quickly
converge on the directions here. Adding user@ list as well...
Cos
On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> Thanks Jay,
> A screencast is something we have had on our todo list; we can easily
> prioritize a bigtop example as the first of a few examples we were
planning
> to do.
>
> As far as the demo, some dates we had in mind were Wednesday or Thursday
> next week.
> I guess the easiest way to see who is interested is a reply/+1
> with Wednesday or Thursday; we then can find exact times.
> We are happy to schedule multiple demos at a few different days/times.
> -Rich
>
>
> ----- Original Message -----
> From:
> dev@bigtop.apache.org
>
> To:
> "dev@bigtop.apache.org" <de...@bigtop.apache.org>
> Cc:
>
> Sent:
> Wed, 25 Feb 2015 12:32:59 -0500
> Subject:
> Re: polishing off bigtop : SSH Provisioning to the cloud
>
>
> Hi richard, also maybe even a youtube video of Devops TK deploying BigTop
> would be awesome !
>
> On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com>
wrote:
>
> > I think might have a solution (dtk.io) that we have worked on that can
> > help
> > solve these problems. We have been working the last four years on the
> > problem of deploying complex clusters. It provides a simple DSL for
> > describing a rich set of topologies (eg simple master slave topologies,
HA
> > variants, security variants, monitoring and centralized logging
> > extensions). It provides one click deployment from the selected cluster
> > topology. It also provides a simple workflow DSL to enable finer grain
> > control on order of execution.
> >
> > We are in the process now of refining a plan to open source it
> >
> > Our system was built to work on top of user's existing configuration
> > assets. The initial focus has been on Puppet, but we are expanding now
for
> > docker support and using arbitrary scripts
> >
> > For last two years we have been using some of the Bigtop puppet modules
> for
> > our big data deployments as well as in concert with modules such as
kafka,
> > storm, accumulo , opentsdb ,.. We are in process now of converting to
use
> > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
> > leverage the groovy and gradle test work so one can bring up cluster
stage
> > by stage with smoke tests after every stage
> >
> > Think it would be straight forward so this can also plug on top of
Bigtop
> > vagrant work; we have a plugable iaas architecture; we currently support
> > ec2 and "managed servers" (we have focused here on bare metal); so
vagrant
> > would extend reach to virtual box and its other iaas providers.
> >
> > I think there is potential for great synergy; best way to show would be
> > through
> > goto meeting demo(s) and answering any detailed questions. We can do
this
> > next week or the following at one or a few sessions. Will shortly send
> > proposed dates/times
> > - Rich
> >
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> > <de...@bigtop.apache.org>
> > Cc:
> >
> > Sent:
> > Wed, 25 Feb 2015 17:24:41 +0800
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud.
> >
> >
> > Cluster deployment and management are important things to make our work
> > finally adopted by users. From my point of view I think there are two
> > things we can improve further:
> > * user friendly cluster auto deployment
> > (we have puppet recipes, but users still need to manually ship puppet
> > code and configurations)
> > * topology layout
> > (for example, specify zookeeper to be deployed on node 3~5)
> >
> > BIGTOP-1702 has a great potential to make cluster deployment more user
> > friendly.
> > That is users don't need to deal with those trivial things like ship
> bigtop
> > puppet code and hiera configurations or install those prerequisites all
> > over the cluster.
> >
> > To answer cos's question in BIGTOP-1702:
> >
> > although I am not sure what you mean by
> > link those managed servers for further provisioning
> >
> > According to the readme of vagrant-managed-servers, vagrant up and
destroy
> > are re-interpreted as "linking//unlinking". After those servers are
> > linked, theoretically, we can directly leverage our Vagrantfile
> > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
> > So, this definitely will take advantage of existing puppet recipes.
> >
> > And here's my thought on following questions:
> >
> > - how that topology will be specified?
> > - how it is going to be compatible with what we have right now? Not
like I
> > try to preserve what we have right now, as there's not much to preserve.
> > We don't really have a way to desribe a cluster's topology at the
> > moment anyway.
> > - in case we need to describe more complex topology - how would be done?
> >
> > I don't think BIGTOP-1702 support topology settings. Regarding to the
> > feature, I think we should support topology specification in our puppet
> > recipes. We currently do not support this yet, but I believe Michael is
> > heading toward this way.
> >
> >
> > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> >
> > > I think it is a great idea!
> > >
> > > And, indeed, it's just a very small step - make it user-friendly ;)
> > >
> > > I believe what you're proposing below is a much better way to achieve
> > what
> > > I've been trying by wrapping Puppet and content distribution into
Gradle
> > > (you
> > > referred to it 'our own deployer' below). That was a bad idea, in
> > > retrospect.
> > > And for sure - we need the functionality of this kind: we need
something
> > > that
> > > let anyone to deploy a cluster to a pre-provisioned nodes quickly
> without
> > > additional troubles!
> > >
> > > I am also a big fan of getting rid of lengthy and at times cryptic
pages
> > > explaining the installation process.
> > >
> > > Now, the solution you have in mind will be helping with a typical
> cluster
> > > topology like we are usually setting up with head_node, and other flat
> > > things
> > > of this nature, right? In other words - a simple, "traditional"
cluster
> > > layout? If so, may I ask a few questions:
> > > - how that topology will be specified?
> > > - how it is going to be compatible with what we have right now? Not
like
> > I
> > > try to preserve what we have right now, as there's not much to
preserve.
> > > We don't really have a way to desribe a cluster's topology at the
> > > moment anyway.
> > > - in case we need to describe more complex topology - how would be
done?
> > >
> > > I also presume, that the proposed solution will take advantage of
> > existing
> > > Puppet recipes, right?
> > >
> > > Thanks in advance for your answers!
> > > Cos
> > >
> > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > Hi folks.
> > > >
> > > > This last year we've made a lot of progress,
> > > > - we can now easily build bigtop,
> > > > - and we can test a fresh build in output/ it in vagrant boxes
locally
> > > >
> > > > Whats next?
> > > >
> > > > In my mind, there is only one last step, to make bigtop a consumer
> > facing
> > > > product: We need to make it so you can simply provision a cluster in
> > the
> > > > cloud.
> > > > As you may recall, David (student at RPI, intern at analytics
company
> > > last
> > > > summer), asked how to do this, and spent about three weeks on it.
> > > Recently
> > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
University,
> > and
> > > > former Yahoo researchers, also asked me the same thing.
> > > >
> > > > Alternative?
> > > >
> > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so
> > on,
> > > > which mostly are an untested, human readable version of our vagrant
> > > > recipes. yikes !
> > > >
> > > > How to implement a ssh cloud bigtop installation ?
> > > >
> > > > We could roll our own deployer, but in doing so, we would have to
make
> > > > semantics for:
> > > >
> > > > - possible need for machiene reboots
> > > > - syncing local folders to remote folders
> > > > - installing (and reinstalling)
> > > >
> > > > luckily, our buddy Vagrant already does this for us (respectively ,
> > with
> > > > the "reload","synced.folder", and "up" options).
> > > >
> > > > Additionally, *** I THINK *** we can literally use the exact same
> > vagrant
> > > > recipes which we are already using to test --- so we will have a
great
> > > user
> > > > experience, and really easy to reproduce bugs and test deployments.
> > > >
> > > >
> > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > > suggestions
> > > > on the implementation or feature? I think it will be easy, but also,
> it
> > > > will be one of the most powerful things we add to bigtop, basically
> > > > allowing users to easily use the system... maybe even too easy :)
> > > >
> > > > --
> > > > jay vyas
> > >
> >
>
>
>
> --
> jay vyas
Re: polishing off bigtop : SSH Provisioning to the cloud
Posted by ri...@reactor8.com.
Evans
We are not set up to record tomorrow's session, but can record
something shortly. Also happy to do a demo that meets your time zone;
so if interested please suggest some times that work and we can
accommodate
- Rich
----- Original Message -----
From: dev@bigtop.apache.org
To:
Cc:
Sent:Thu, 5 Mar 2015 14:21:35 +0800
Subject:Re: polishing off bigtop : SSH Provisioning to the cloud
Hi Richard,
Thanks for setting up this.
I can't join the meeting because of time zone difference.
I think it would be great to have this recorded:)
Evans
2015-03-05 13:15 GMT+08:00 Konstantin Boudnik :
> Guys, just want to remind everyone that we have the DTK demo
tomorrow 10
> PST
> - should be great!
>
> Cos
>
> On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote:
> > We set up a goto meeting for 10:00AM PST on Thursday March 5th
> >
> > *Bigtop - DevOps Toolkit (DTK) Demo*
> > Thu, Mar 5, 10:00 AM Pacific Standard Time
> >
> >
> > - Please join my meeting from your computer, tablet or
smartphone.
> > https://global.gotomeeting.com/join/655931173
> > - You can also dial in using your phone.
> > United States (Long distance): +1 (571) 317-3122
> > *Access Code:* 655-931-173
> > More phone numbers:
> > https://global.gotomeeting.com/655931173/numbersdisplay.html
> >
> > If interested, but this day or time does not work, please send a
note and
> > we can set up additional day/time(s) too.
> >
> > We will have a few high level slides and then have a demo that
shows how
> we
> > currently use Bigtop and how this can be naturally extended .to
provide a
> > "one click" solution to deploy a rich variety of bigtop stacks
with
> > different mixtures of services and different topologies that
plugs on top
> > of the existing bigtop deployment work
> >
> > As far as a screencast, we wanted our first one to be a short
focused
> > screencast; so that will be a follow up that we will put together
shortly
> > - Rich
> >
> >
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> >
> > Cc:
> >
> > Sent:
> > Sat, 28 Feb 2015 01:31:10 +0000
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud
> >
> >
> > Either of the days is open for me: would love to have a demo.
Perhaps it
> can
> > be recorded to be used as a screencast later on? ;)
> >
> > It'd be great to have as many people as possible on that, so we
can
> quickly
> > converge on the directions here. Adding user@ list as well...
> >
> > Cos
> >
> > On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> > > Thanks Jay,
> > > A screencast is something we have had on our todo list; we can
easily
> > > prioritize a bigtop example as the first of a few examples we
were
> > planning
> > > to do.
> > >
> > > As far as the demo, some dates we had in mind were Wednesday or
> Thursday
> > > next week.
> > > I guess the easiest way to see who is interested is a reply/+1
> > > with Wednesday or Thursday; we then can find exact times.
> > > We are happy to schedule multiple demos at a few different
days/times.
> > > -Rich
> > >
> > >
> > > ----- Original Message -----
> > > From:
> > > dev@bigtop.apache.org
> > >
> > > To:
> > > "dev@bigtop.apache.org"
> > > Cc:
> > >
> > > Sent:
> > > Wed, 25 Feb 2015 12:32:59 -0500
> > > Subject:
> > > Re: polishing off bigtop : SSH Provisioning to the cloud
> > >
> > >
> > > Hi richard, also maybe even a youtube video of Devops TK
deploying
> BigTop
> > > would be awesome !
> > >
> > > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin
> > wrote:
> > >
> > > > I think might have a solution (dtk.io) that we have worked on
that
> can
> > > > help
> > > > solve these problems. We have been working the last four
years on the
> > > > problem of deploying complex clusters. It provides a simple
DSL for
> > > > describing a rich set of topologies (eg simple master slave
> topologies,
> > HA
> > > > variants, security variants, monitoring and centralized
logging
> > > > extensions). It provides one click deployment from the
selected
> cluster
> > > > topology. It also provides a simple workflow DSL to enable
finer
> grain
> > > > control on order of execution.
> > > >
> > > > We are in the process now of refining a plan to open source
it
> > > >
> > > > Our system was built to work on top of user's existing
configuration
> > > > assets. The initial focus has been on Puppet, but we are
expanding
> now
> > for
> > > > docker support and using arbitrary scripts
> > > >
> > > > For last two years we have been using some of the Bigtop
puppet
> modules
> > > for
> > > > our big data deployments as well as in concert with modules
such as
> > kafka,
> > > > storm, accumulo , opentsdb ,. We are in process now of
converting to
> > use
> > > > the new Bigtop Puppet 3.x hiera modules. We also just put in
ability
> to
> > > > leverage the groovy and gradle test work so one can bring up
cluster
> > stage
> > > > by stage with smoke tests after every stage
> > > >
> > > > Think it would be straight forward so this can also plug on
top of
> > Bigtop
> > > > vagrant work; we have a plugable iaas architecture; we
currently
> support
> > > > ec2 and "managed servers" (we have focused here on bare
metal); so
> > vagrant
> > > > would extend reach to virtual box and its other iaas
providers.
> > > >
> > > > I think there is potential for great synergy; best way to
show would
> be
> > > > through
> > > > goto meeting demo(s) and answering any detailed questions. We
can do
> > this
> > > > next week or the following at one or a few sessions. Will
shortly
> send
> > > > proposed dates/times
> > > > - Rich
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From:
> > > > dev@bigtop.apache.org
> > > >
> > > > To:
> > > >
> > > > Cc:
> > > >
> > > > Sent:
> > > > Wed, 25 Feb 2015 17:24:41 +0800
> > > > Subject:
> > > > Re: polishing off bigtop : SSH Provisioning to the cloud.
> > > >
> > > >
> > > > Cluster deployment and management are important things to
make our
> work
> > > > finally adopted by users. From my point of view I think there
are two
> > > > things we can improve further:
> > > > * user friendly cluster auto deployment
> > > > (we have puppet recipes, but users still need to manually
ship puppet
> > > > code and configurations)
> > > > * topology layout
> > > > (for example, specify zookeeper to be deployed on node 3~5)
> > > >
> > > > BIGTOP-1702 has a great potential to make cluster deployment
more
> user
> > > > friendly.
> > > > That is users don't need to deal with those trivial things
like ship
> > > bigtop
> > > > puppet code and hiera configurations or install those
prerequisites
> all
> > > > over the cluster.
> > > >
> > > > To answer cos's question in BIGTOP-1702:
> > > >
> > > > although I am not sure what you mean by
> > > > link those managed servers for further provisioning
> > > >
> > > > According to the readme of vagrant-managed-servers, vagrant
up and
> > destroy
> > > > are re-interpreted as "linking//unlinking". After those
servers are
> > > > linked, theoretically, we can directly leverage our
Vagrantfile
> > > > in "vagrant-puppet-vm" to do the whole cluster deployment by
> one-click.
> > > > So, this definitely will take advantage of existing puppet
recipes.
> > > >
> > > > And here's my thought on following questions:
> > > >
> > > > - how that topology will be specified?
> > > > - how it is going to be compatible with what we have right
now? Not
> > like I
> > > > try to preserve what we have right now, as there's not much
to
> preserve.
> > > > We don't really have a way to desribe a cluster's topology at
the
> > > > moment anyway.
> > > > - in case we need to describe more complex topology - how
would be
> done?
> > > >
> > > > I don't think BIGTOP-1702 support topology settings.
Regarding to the
> > > > feature, I think we should support topology specification in
our
> puppet
> > > > recipes. We currently do not support this yet, but I believe
Michael
> is
> > > > heading toward this way.
> > > >
> > > >
> > > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik :
> > > >
> > > > > I think it is a great idea!
> > > > >
> > > > > And, indeed, it's just a very small step - make it
user-friendly ;)
> > > > >
> > > > > I believe what you're proposing below is a much better way
to
> achieve
> > > > what
> > > > > I've been trying by wrapping Puppet and content
distribution into
> > Gradle
> > > > > (you
> > > > > referred to it 'our own deployer' below) That was a bad
idea, in
> > > > > retrospect.
> > > > > And for sure - we need the functionality of this kind: we
need
> > something
> > > > > that
> > > > > let anyone to deploy a cluster to a pre-provisioned nodes
quickly
> > > without
> > > > > additional troubles!
> > > > >
> > > > > I am also a big fan of getting rid of lengthy and at times
cryptic
> > pages
> > > > > explaining the installation process.
> > > > >
> > > > > Now, the solution you have in mind will be helping with a
typical
> > > cluster
> > > > > topology like we are usually setting up with head_node, and
other
> flat
> > > > > things
> > > > > of this nature, right? In other words - a simple,
"traditional"
> > cluster
> > > > > layout? If so, may I ask a few questions:
> > > > > - how that topology will be specified?
> > > > > - how it is going to be compatible with what we have right
now? Not
> > like
> > > > I
> > > > > try to preserve what we have right now, as there's not much
to
> > preserve.
> > > > > We don't really have a way to desribe a cluster's topology
at the
> > > > > moment anyway.
> > > > > - in case we need to describe more complex topology - how
would be
> > done?
> > > > >
> > > > > I also presume, that the proposed solution will take
advantage of
> > > > existing
> > > > > Puppet recipes, right?
> > > > >
> > > > > Thanks in advance for your answers!
> > > > > Cos
> > > > >
> > > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > > > Hi folks.
> > > > > >
> > > > > > This last year we've made a lot of progress,
> > > > > > - we can now easily build bigtop,
> > > > > > - and we can test a fresh build in output/ it in vagrant
boxes
> > locally
> > > > > >
> > > > > > Whats next?
> > > > > >
> > > > > > In my mind, there is only one last step, to make bigtop a
> consumer
> > > > facing
> > > > > > product: We need to make it so you can simply provision a
> cluster in
> > > > the
> > > > > > cloud.
> > > > > > As you may recall, David (student at RPI, intern at
analytics
> > company
> > > > > last
> > > > > > summer), asked how to do this, and spent about three
weeks on it.
> > > > > Recently
> > > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
> > University,
> > > > and
> > > > > > former Yahoo researchers, also asked me the same thing.
> > > > > >
> > > > > > Alternative?
> > > > > >
> > > > > > Well we currently these docs about how to setup 0.7.0,
0.8.0,
> and so
> > > > on,
> > > > > > which mostly are an untested, human readable version of
our
> vagrant
> > > > > > recipes. yikes !
> > > > > >
> > > > > > How to implement a ssh cloud bigtop installation ?
> > > > > >
> > > > > > We could roll our own deployer, but in doing so, we would
have to
> > make
> > > > > > semantics for:
> > > > > >
> > > > > > - possible need for machiene reboots
> > > > > > - syncing local folders to remote folders
> > > > > > - installing (and reinstalling)
> > > > > >
> > > > > > luckily, our buddy Vagrant already does this for us
> (respectively ,
> > > > with
> > > > > > the "reload","synced.folder", and "up" options).
> > > > > >
> > > > > > Additionally, *** I THINK *** we can literally use the
exact same
> > > > vagrant
> > > > > > recipes which we are already using to test --- so we will
have a
> > great
> > > > > user
> > > > > > experience, and really easy to reproduce bugs and test
> deployments.
> > > > > >
> > > > > >
> > > > > > I';l hash out details in BIGTOP-1702 , any thoughts,
questions,
> > > > > suggestions
> > > > > > on the implementation or feature? I think it will be
easy, but
> also,
> > > it
> > > > > > will be one of the most powerful things we add to bigtop,
> basically
> > > > > > allowing users to easily use the system... maybe even too
easy :)
> > > > > >
> > > > > > --
> > > > > > jay vyas
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > jay vyas
>
Re: polishing off bigtop : SSH Provisioning to the cloud
Posted by Evans Ye <ev...@apache.org>.
Hi Richard,
Thanks for setting up this.
I can't join the meeting because of time zone difference.
I think it would be great to have this recorded:)
Evans
2015-03-05 13:15 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> Guys, just want to remind everyone that we have the DTK demo tomorrow 10
> PST
> - should be great!
>
> Cos
>
> On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote:
> > We set up a goto meeting for 10:00AM PST on Thursday March 5th
> >
> > *Bigtop - DevOps Toolkit (DTK) Demo*
> > Thu, Mar 5, 10:00 AM Pacific Standard Time
> >
> >
> > - Please join my meeting from your computer, tablet or smartphone.
> > https://global.gotomeeting.com/join/655931173
> > - You can also dial in using your phone.
> > United States (Long distance): +1 (571) 317-3122
> > *Access Code:* 655-931-173
> > More phone numbers:
> > https://global.gotomeeting.com/655931173/numbersdisplay.html
> >
> > If interested, but this day or time does not work, please send a note and
> > we can set up additional day/time(s) too.
> >
> > We will have a few high level slides and then have a demo that shows how
> we
> > currently use Bigtop and how this can be naturally extended .to provide a
> > "one click" solution to deploy a rich variety of bigtop stacks with
> > different mixtures of services and different topologies that plugs on top
> > of the existing bigtop deployment work
> >
> > As far as a screencast, we wanted our first one to be a short focused
> > screencast; so that will be a follow up that we will put together shortly
> > - Rich
> >
> >
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> > <de...@bigtop.apache.org>
> > Cc:
> > <us...@bigtop.apache.org>
> > Sent:
> > Sat, 28 Feb 2015 01:31:10 +0000
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud
> >
> >
> > Either of the days is open for me: would love to have a demo. Perhaps it
> can
> > be recorded to be used as a screencast later on? ;)
> >
> > It'd be great to have as many people as possible on that, so we can
> quickly
> > converge on the directions here. Adding user@ list as well...
> >
> > Cos
> >
> > On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> > > Thanks Jay,
> > > A screencast is something we have had on our todo list; we can easily
> > > prioritize a bigtop example as the first of a few examples we were
> > planning
> > > to do.
> > >
> > > As far as the demo, some dates we had in mind were Wednesday or
> Thursday
> > > next week.
> > > I guess the easiest way to see who is interested is a reply/+1
> > > with Wednesday or Thursday; we then can find exact times.
> > > We are happy to schedule multiple demos at a few different days/times.
> > > -Rich
> > >
> > >
> > > ----- Original Message -----
> > > From:
> > > dev@bigtop.apache.org
> > >
> > > To:
> > > "dev@bigtop.apache.org" <de...@bigtop.apache.org>
> > > Cc:
> > >
> > > Sent:
> > > Wed, 25 Feb 2015 12:32:59 -0500
> > > Subject:
> > > Re: polishing off bigtop : SSH Provisioning to the cloud
> > >
> > >
> > > Hi richard, also maybe even a youtube video of Devops TK deploying
> BigTop
> > > would be awesome !
> > >
> > > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com>
> > wrote:
> > >
> > > > I think might have a solution (dtk.io) that we have worked on that
> can
> > > > help
> > > > solve these problems. We have been working the last four years on the
> > > > problem of deploying complex clusters. It provides a simple DSL for
> > > > describing a rich set of topologies (eg simple master slave
> topologies,
> > HA
> > > > variants, security variants, monitoring and centralized logging
> > > > extensions). It provides one click deployment from the selected
> cluster
> > > > topology. It also provides a simple workflow DSL to enable finer
> grain
> > > > control on order of execution.
> > > >
> > > > We are in the process now of refining a plan to open source it
> > > >
> > > > Our system was built to work on top of user's existing configuration
> > > > assets. The initial focus has been on Puppet, but we are expanding
> now
> > for
> > > > docker support and using arbitrary scripts
> > > >
> > > > For last two years we have been using some of the Bigtop puppet
> modules
> > > for
> > > > our big data deployments as well as in concert with modules such as
> > kafka,
> > > > storm, accumulo , opentsdb ,.. We are in process now of converting to
> > use
> > > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability
> to
> > > > leverage the groovy and gradle test work so one can bring up cluster
> > stage
> > > > by stage with smoke tests after every stage
> > > >
> > > > Think it would be straight forward so this can also plug on top of
> > Bigtop
> > > > vagrant work; we have a plugable iaas architecture; we currently
> support
> > > > ec2 and "managed servers" (we have focused here on bare metal); so
> > vagrant
> > > > would extend reach to virtual box and its other iaas providers.
> > > >
> > > > I think there is potential for great synergy; best way to show would
> be
> > > > through
> > > > goto meeting demo(s) and answering any detailed questions. We can do
> > this
> > > > next week or the following at one or a few sessions. Will shortly
> send
> > > > proposed dates/times
> > > > - Rich
> > > >
> > > >
> > > >
> > > > ----- Original Message -----
> > > > From:
> > > > dev@bigtop.apache.org
> > > >
> > > > To:
> > > > <de...@bigtop.apache.org>
> > > > Cc:
> > > >
> > > > Sent:
> > > > Wed, 25 Feb 2015 17:24:41 +0800
> > > > Subject:
> > > > Re: polishing off bigtop : SSH Provisioning to the cloud.
> > > >
> > > >
> > > > Cluster deployment and management are important things to make our
> work
> > > > finally adopted by users. From my point of view I think there are two
> > > > things we can improve further:
> > > > * user friendly cluster auto deployment
> > > > (we have puppet recipes, but users still need to manually ship puppet
> > > > code and configurations)
> > > > * topology layout
> > > > (for example, specify zookeeper to be deployed on node 3~5)
> > > >
> > > > BIGTOP-1702 has a great potential to make cluster deployment more
> user
> > > > friendly.
> > > > That is users don't need to deal with those trivial things like ship
> > > bigtop
> > > > puppet code and hiera configurations or install those prerequisites
> all
> > > > over the cluster.
> > > >
> > > > To answer cos's question in BIGTOP-1702:
> > > >
> > > > although I am not sure what you mean by
> > > > link those managed servers for further provisioning
> > > >
> > > > According to the readme of vagrant-managed-servers, vagrant up and
> > destroy
> > > > are re-interpreted as "linking//unlinking". After those servers are
> > > > linked, theoretically, we can directly leverage our Vagrantfile
> > > > in "vagrant-puppet-vm" to do the whole cluster deployment by
> one-click.
> > > > So, this definitely will take advantage of existing puppet recipes.
> > > >
> > > > And here's my thought on following questions:
> > > >
> > > > - how that topology will be specified?
> > > > - how it is going to be compatible with what we have right now? Not
> > like I
> > > > try to preserve what we have right now, as there's not much to
> preserve.
> > > > We don't really have a way to desribe a cluster's topology at the
> > > > moment anyway.
> > > > - in case we need to describe more complex topology - how would be
> done?
> > > >
> > > > I don't think BIGTOP-1702 support topology settings. Regarding to the
> > > > feature, I think we should support topology specification in our
> puppet
> > > > recipes. We currently do not support this yet, but I believe Michael
> is
> > > > heading toward this way.
> > > >
> > > >
> > > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > > >
> > > > > I think it is a great idea!
> > > > >
> > > > > And, indeed, it's just a very small step - make it user-friendly ;)
> > > > >
> > > > > I believe what you're proposing below is a much better way to
> achieve
> > > > what
> > > > > I've been trying by wrapping Puppet and content distribution into
> > Gradle
> > > > > (you
> > > > > referred to it 'our own deployer' below). That was a bad idea, in
> > > > > retrospect.
> > > > > And for sure - we need the functionality of this kind: we need
> > something
> > > > > that
> > > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly
> > > without
> > > > > additional troubles!
> > > > >
> > > > > I am also a big fan of getting rid of lengthy and at times cryptic
> > pages
> > > > > explaining the installation process.
> > > > >
> > > > > Now, the solution you have in mind will be helping with a typical
> > > cluster
> > > > > topology like we are usually setting up with head_node, and other
> flat
> > > > > things
> > > > > of this nature, right? In other words - a simple, "traditional"
> > cluster
> > > > > layout? If so, may I ask a few questions:
> > > > > - how that topology will be specified?
> > > > > - how it is going to be compatible with what we have right now? Not
> > like
> > > > I
> > > > > try to preserve what we have right now, as there's not much to
> > preserve.
> > > > > We don't really have a way to desribe a cluster's topology at the
> > > > > moment anyway.
> > > > > - in case we need to describe more complex topology - how would be
> > done?
> > > > >
> > > > > I also presume, that the proposed solution will take advantage of
> > > > existing
> > > > > Puppet recipes, right?
> > > > >
> > > > > Thanks in advance for your answers!
> > > > > Cos
> > > > >
> > > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > > > Hi folks.
> > > > > >
> > > > > > This last year we've made a lot of progress,
> > > > > > - we can now easily build bigtop,
> > > > > > - and we can test a fresh build in output/ it in vagrant boxes
> > locally
> > > > > >
> > > > > > Whats next?
> > > > > >
> > > > > > In my mind, there is only one last step, to make bigtop a
> consumer
> > > > facing
> > > > > > product: We need to make it so you can simply provision a
> cluster in
> > > > the
> > > > > > cloud.
> > > > > > As you may recall, David (student at RPI, intern at analytics
> > company
> > > > > last
> > > > > > summer), asked how to do this, and spent about three weeks on it.
> > > > > Recently
> > > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
> > University,
> > > > and
> > > > > > former Yahoo researchers, also asked me the same thing.
> > > > > >
> > > > > > Alternative?
> > > > > >
> > > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0,
> and so
> > > > on,
> > > > > > which mostly are an untested, human readable version of our
> vagrant
> > > > > > recipes. yikes !
> > > > > >
> > > > > > How to implement a ssh cloud bigtop installation ?
> > > > > >
> > > > > > We could roll our own deployer, but in doing so, we would have to
> > make
> > > > > > semantics for:
> > > > > >
> > > > > > - possible need for machiene reboots
> > > > > > - syncing local folders to remote folders
> > > > > > - installing (and reinstalling)
> > > > > >
> > > > > > luckily, our buddy Vagrant already does this for us
> (respectively ,
> > > > with
> > > > > > the "reload","synced.folder", and "up" options).
> > > > > >
> > > > > > Additionally, *** I THINK *** we can literally use the exact same
> > > > vagrant
> > > > > > recipes which we are already using to test --- so we will have a
> > great
> > > > > user
> > > > > > experience, and really easy to reproduce bugs and test
> deployments.
> > > > > >
> > > > > >
> > > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > > > > suggestions
> > > > > > on the implementation or feature? I think it will be easy, but
> also,
> > > it
> > > > > > will be one of the most powerful things we add to bigtop,
> basically
> > > > > > allowing users to easily use the system... maybe even too easy :)
> > > > > >
> > > > > > --
> > > > > > jay vyas
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > jay vyas
>
Re: polishing off bigtop : SSH Provisioning to the cloud
Posted by Konstantin Boudnik <co...@apache.org>.
Guys, just want to remind everyone that we have the DTK demo tomorrow 10 PST
- should be great!
Cos
On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote:
> We set up a goto meeting for 10:00AM PST on Thursday March 5th
>
> *Bigtop - DevOps Toolkit (DTK) Demo*
> Thu, Mar 5, 10:00 AM Pacific Standard Time
>
>
> - Please join my meeting from your computer, tablet or smartphone.
> https://global.gotomeeting.com/join/655931173
> - You can also dial in using your phone.
> United States (Long distance): +1 (571) 317-3122
> *Access Code:* 655-931-173
> More phone numbers:
> https://global.gotomeeting.com/655931173/numbersdisplay.html
>
> If interested, but this day or time does not work, please send a note and
> we can set up additional day/time(s) too.
>
> We will have a few high level slides and then have a demo that shows how we
> currently use Bigtop and how this can be naturally extended .to provide a
> "one click" solution to deploy a rich variety of bigtop stacks with
> different mixtures of services and different topologies that plugs on top
> of the existing bigtop deployment work
>
> As far as a screencast, we wanted our first one to be a short focused
> screencast; so that will be a follow up that we will put together shortly
> - Rich
>
>
>
>
> ----- Original Message -----
> From:
> dev@bigtop.apache.org
>
> To:
> <de...@bigtop.apache.org>
> Cc:
> <us...@bigtop.apache.org>
> Sent:
> Sat, 28 Feb 2015 01:31:10 +0000
> Subject:
> Re: polishing off bigtop : SSH Provisioning to the cloud
>
>
> Either of the days is open for me: would love to have a demo. Perhaps it can
> be recorded to be used as a screencast later on? ;)
>
> It'd be great to have as many people as possible on that, so we can quickly
> converge on the directions here. Adding user@ list as well...
>
> Cos
>
> On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> > Thanks Jay,
> > A screencast is something we have had on our todo list; we can easily
> > prioritize a bigtop example as the first of a few examples we were
> planning
> > to do.
> >
> > As far as the demo, some dates we had in mind were Wednesday or Thursday
> > next week.
> > I guess the easiest way to see who is interested is a reply/+1
> > with Wednesday or Thursday; we then can find exact times.
> > We are happy to schedule multiple demos at a few different days/times.
> > -Rich
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> > "dev@bigtop.apache.org" <de...@bigtop.apache.org>
> > Cc:
> >
> > Sent:
> > Wed, 25 Feb 2015 12:32:59 -0500
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud
> >
> >
> > Hi richard, also maybe even a youtube video of Devops TK deploying BigTop
> > would be awesome !
> >
> > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com>
> wrote:
> >
> > > I think might have a solution (dtk.io) that we have worked on that can
> > > help
> > > solve these problems. We have been working the last four years on the
> > > problem of deploying complex clusters. It provides a simple DSL for
> > > describing a rich set of topologies (eg simple master slave topologies,
> HA
> > > variants, security variants, monitoring and centralized logging
> > > extensions). It provides one click deployment from the selected cluster
> > > topology. It also provides a simple workflow DSL to enable finer grain
> > > control on order of execution.
> > >
> > > We are in the process now of refining a plan to open source it
> > >
> > > Our system was built to work on top of user's existing configuration
> > > assets. The initial focus has been on Puppet, but we are expanding now
> for
> > > docker support and using arbitrary scripts
> > >
> > > For last two years we have been using some of the Bigtop puppet modules
> > for
> > > our big data deployments as well as in concert with modules such as
> kafka,
> > > storm, accumulo , opentsdb ,.. We are in process now of converting to
> use
> > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
> > > leverage the groovy and gradle test work so one can bring up cluster
> stage
> > > by stage with smoke tests after every stage
> > >
> > > Think it would be straight forward so this can also plug on top of
> Bigtop
> > > vagrant work; we have a plugable iaas architecture; we currently support
> > > ec2 and "managed servers" (we have focused here on bare metal); so
> vagrant
> > > would extend reach to virtual box and its other iaas providers.
> > >
> > > I think there is potential for great synergy; best way to show would be
> > > through
> > > goto meeting demo(s) and answering any detailed questions. We can do
> this
> > > next week or the following at one or a few sessions. Will shortly send
> > > proposed dates/times
> > > - Rich
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From:
> > > dev@bigtop.apache.org
> > >
> > > To:
> > > <de...@bigtop.apache.org>
> > > Cc:
> > >
> > > Sent:
> > > Wed, 25 Feb 2015 17:24:41 +0800
> > > Subject:
> > > Re: polishing off bigtop : SSH Provisioning to the cloud.
> > >
> > >
> > > Cluster deployment and management are important things to make our work
> > > finally adopted by users. From my point of view I think there are two
> > > things we can improve further:
> > > * user friendly cluster auto deployment
> > > (we have puppet recipes, but users still need to manually ship puppet
> > > code and configurations)
> > > * topology layout
> > > (for example, specify zookeeper to be deployed on node 3~5)
> > >
> > > BIGTOP-1702 has a great potential to make cluster deployment more user
> > > friendly.
> > > That is users don't need to deal with those trivial things like ship
> > bigtop
> > > puppet code and hiera configurations or install those prerequisites all
> > > over the cluster.
> > >
> > > To answer cos's question in BIGTOP-1702:
> > >
> > > although I am not sure what you mean by
> > > link those managed servers for further provisioning
> > >
> > > According to the readme of vagrant-managed-servers, vagrant up and
> destroy
> > > are re-interpreted as "linking//unlinking". After those servers are
> > > linked, theoretically, we can directly leverage our Vagrantfile
> > > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
> > > So, this definitely will take advantage of existing puppet recipes.
> > >
> > > And here's my thought on following questions:
> > >
> > > - how that topology will be specified?
> > > - how it is going to be compatible with what we have right now? Not
> like I
> > > try to preserve what we have right now, as there's not much to preserve.
> > > We don't really have a way to desribe a cluster's topology at the
> > > moment anyway.
> > > - in case we need to describe more complex topology - how would be done?
> > >
> > > I don't think BIGTOP-1702 support topology settings. Regarding to the
> > > feature, I think we should support topology specification in our puppet
> > > recipes. We currently do not support this yet, but I believe Michael is
> > > heading toward this way.
> > >
> > >
> > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > I think it is a great idea!
> > > >
> > > > And, indeed, it's just a very small step - make it user-friendly ;)
> > > >
> > > > I believe what you're proposing below is a much better way to achieve
> > > what
> > > > I've been trying by wrapping Puppet and content distribution into
> Gradle
> > > > (you
> > > > referred to it 'our own deployer' below). That was a bad idea, in
> > > > retrospect.
> > > > And for sure - we need the functionality of this kind: we need
> something
> > > > that
> > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly
> > without
> > > > additional troubles!
> > > >
> > > > I am also a big fan of getting rid of lengthy and at times cryptic
> pages
> > > > explaining the installation process.
> > > >
> > > > Now, the solution you have in mind will be helping with a typical
> > cluster
> > > > topology like we are usually setting up with head_node, and other flat
> > > > things
> > > > of this nature, right? In other words - a simple, "traditional"
> cluster
> > > > layout? If so, may I ask a few questions:
> > > > - how that topology will be specified?
> > > > - how it is going to be compatible with what we have right now? Not
> like
> > > I
> > > > try to preserve what we have right now, as there's not much to
> preserve.
> > > > We don't really have a way to desribe a cluster's topology at the
> > > > moment anyway.
> > > > - in case we need to describe more complex topology - how would be
> done?
> > > >
> > > > I also presume, that the proposed solution will take advantage of
> > > existing
> > > > Puppet recipes, right?
> > > >
> > > > Thanks in advance for your answers!
> > > > Cos
> > > >
> > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > > Hi folks.
> > > > >
> > > > > This last year we've made a lot of progress,
> > > > > - we can now easily build bigtop,
> > > > > - and we can test a fresh build in output/ it in vagrant boxes
> locally
> > > > >
> > > > > Whats next?
> > > > >
> > > > > In my mind, there is only one last step, to make bigtop a consumer
> > > facing
> > > > > product: We need to make it so you can simply provision a cluster in
> > > the
> > > > > cloud.
> > > > > As you may recall, David (student at RPI, intern at analytics
> company
> > > > last
> > > > > summer), asked how to do this, and spent about three weeks on it.
> > > > Recently
> > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
> University,
> > > and
> > > > > former Yahoo researchers, also asked me the same thing.
> > > > >
> > > > > Alternative?
> > > > >
> > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so
> > > on,
> > > > > which mostly are an untested, human readable version of our vagrant
> > > > > recipes. yikes !
> > > > >
> > > > > How to implement a ssh cloud bigtop installation ?
> > > > >
> > > > > We could roll our own deployer, but in doing so, we would have to
> make
> > > > > semantics for:
> > > > >
> > > > > - possible need for machiene reboots
> > > > > - syncing local folders to remote folders
> > > > > - installing (and reinstalling)
> > > > >
> > > > > luckily, our buddy Vagrant already does this for us (respectively ,
> > > with
> > > > > the "reload","synced.folder", and "up" options).
> > > > >
> > > > > Additionally, *** I THINK *** we can literally use the exact same
> > > vagrant
> > > > > recipes which we are already using to test --- so we will have a
> great
> > > > user
> > > > > experience, and really easy to reproduce bugs and test deployments.
> > > > >
> > > > >
> > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > > > suggestions
> > > > > on the implementation or feature? I think it will be easy, but also,
> > it
> > > > > will be one of the most powerful things we add to bigtop, basically
> > > > > allowing users to easily use the system... maybe even too easy :)
> > > > >
> > > > > --
> > > > > jay vyas
> > > >
> > >
> >
> >
> >
> > --
> > jay vyas
Re: polishing off bigtop : SSH Provisioning to the cloud
Posted by Konstantin Boudnik <co...@apache.org>.
Count me in.
Thanks!
Cos
On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote:
> We set up a goto meeting for 10:00AM PST on Thursday March 5th
>
> *Bigtop - DevOps Toolkit (DTK) Demo*
> Thu, Mar 5, 10:00 AM Pacific Standard Time
>
>
> - Please join my meeting from your computer, tablet or smartphone.
> https://global.gotomeeting.com/join/655931173
> - You can also dial in using your phone.
> United States (Long distance): +1 (571) 317-3122
> *Access Code:* 655-931-173
> More phone numbers:
> https://global.gotomeeting.com/655931173/numbersdisplay.html
>
> If interested, but this day or time does not work, please send a note and
> we can set up additional day/time(s) too.
>
> We will have a few high level slides and then have a demo that shows how we
> currently use Bigtop and how this can be naturally extended .to provide a
> "one click" solution to deploy a rich variety of bigtop stacks with
> different mixtures of services and different topologies that plugs on top
> of the existing bigtop deployment work
>
> As far as a screencast, we wanted our first one to be a short focused
> screencast; so that will be a follow up that we will put together shortly
> - Rich
>
>
>
>
> ----- Original Message -----
> From:
> dev@bigtop.apache.org
>
> To:
> <de...@bigtop.apache.org>
> Cc:
> <us...@bigtop.apache.org>
> Sent:
> Sat, 28 Feb 2015 01:31:10 +0000
> Subject:
> Re: polishing off bigtop : SSH Provisioning to the cloud
>
>
> Either of the days is open for me: would love to have a demo. Perhaps it can
> be recorded to be used as a screencast later on? ;)
>
> It'd be great to have as many people as possible on that, so we can quickly
> converge on the directions here. Adding user@ list as well...
>
> Cos
>
> On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> > Thanks Jay,
> > A screencast is something we have had on our todo list; we can easily
> > prioritize a bigtop example as the first of a few examples we were
> planning
> > to do.
> >
> > As far as the demo, some dates we had in mind were Wednesday or Thursday
> > next week.
> > I guess the easiest way to see who is interested is a reply/+1
> > with Wednesday or Thursday; we then can find exact times.
> > We are happy to schedule multiple demos at a few different days/times.
> > -Rich
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> > "dev@bigtop.apache.org" <de...@bigtop.apache.org>
> > Cc:
> >
> > Sent:
> > Wed, 25 Feb 2015 12:32:59 -0500
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud
> >
> >
> > Hi richard, also maybe even a youtube video of Devops TK deploying BigTop
> > would be awesome !
> >
> > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com>
> wrote:
> >
> > > I think might have a solution (dtk.io) that we have worked on that can
> > > help
> > > solve these problems. We have been working the last four years on the
> > > problem of deploying complex clusters. It provides a simple DSL for
> > > describing a rich set of topologies (eg simple master slave topologies,
> HA
> > > variants, security variants, monitoring and centralized logging
> > > extensions). It provides one click deployment from the selected cluster
> > > topology. It also provides a simple workflow DSL to enable finer grain
> > > control on order of execution.
> > >
> > > We are in the process now of refining a plan to open source it
> > >
> > > Our system was built to work on top of user's existing configuration
> > > assets. The initial focus has been on Puppet, but we are expanding now
> for
> > > docker support and using arbitrary scripts
> > >
> > > For last two years we have been using some of the Bigtop puppet modules
> > for
> > > our big data deployments as well as in concert with modules such as
> kafka,
> > > storm, accumulo , opentsdb ,.. We are in process now of converting to
> use
> > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
> > > leverage the groovy and gradle test work so one can bring up cluster
> stage
> > > by stage with smoke tests after every stage
> > >
> > > Think it would be straight forward so this can also plug on top of
> Bigtop
> > > vagrant work; we have a plugable iaas architecture; we currently support
> > > ec2 and "managed servers" (we have focused here on bare metal); so
> vagrant
> > > would extend reach to virtual box and its other iaas providers.
> > >
> > > I think there is potential for great synergy; best way to show would be
> > > through
> > > goto meeting demo(s) and answering any detailed questions. We can do
> this
> > > next week or the following at one or a few sessions. Will shortly send
> > > proposed dates/times
> > > - Rich
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From:
> > > dev@bigtop.apache.org
> > >
> > > To:
> > > <de...@bigtop.apache.org>
> > > Cc:
> > >
> > > Sent:
> > > Wed, 25 Feb 2015 17:24:41 +0800
> > > Subject:
> > > Re: polishing off bigtop : SSH Provisioning to the cloud.
> > >
> > >
> > > Cluster deployment and management are important things to make our work
> > > finally adopted by users. From my point of view I think there are two
> > > things we can improve further:
> > > * user friendly cluster auto deployment
> > > (we have puppet recipes, but users still need to manually ship puppet
> > > code and configurations)
> > > * topology layout
> > > (for example, specify zookeeper to be deployed on node 3~5)
> > >
> > > BIGTOP-1702 has a great potential to make cluster deployment more user
> > > friendly.
> > > That is users don't need to deal with those trivial things like ship
> > bigtop
> > > puppet code and hiera configurations or install those prerequisites all
> > > over the cluster.
> > >
> > > To answer cos's question in BIGTOP-1702:
> > >
> > > although I am not sure what you mean by
> > > link those managed servers for further provisioning
> > >
> > > According to the readme of vagrant-managed-servers, vagrant up and
> destroy
> > > are re-interpreted as "linking//unlinking". After those servers are
> > > linked, theoretically, we can directly leverage our Vagrantfile
> > > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
> > > So, this definitely will take advantage of existing puppet recipes.
> > >
> > > And here's my thought on following questions:
> > >
> > > - how that topology will be specified?
> > > - how it is going to be compatible with what we have right now? Not
> like I
> > > try to preserve what we have right now, as there's not much to preserve.
> > > We don't really have a way to desribe a cluster's topology at the
> > > moment anyway.
> > > - in case we need to describe more complex topology - how would be done?
> > >
> > > I don't think BIGTOP-1702 support topology settings. Regarding to the
> > > feature, I think we should support topology specification in our puppet
> > > recipes. We currently do not support this yet, but I believe Michael is
> > > heading toward this way.
> > >
> > >
> > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > I think it is a great idea!
> > > >
> > > > And, indeed, it's just a very small step - make it user-friendly ;)
> > > >
> > > > I believe what you're proposing below is a much better way to achieve
> > > what
> > > > I've been trying by wrapping Puppet and content distribution into
> Gradle
> > > > (you
> > > > referred to it 'our own deployer' below). That was a bad idea, in
> > > > retrospect.
> > > > And for sure - we need the functionality of this kind: we need
> something
> > > > that
> > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly
> > without
> > > > additional troubles!
> > > >
> > > > I am also a big fan of getting rid of lengthy and at times cryptic
> pages
> > > > explaining the installation process.
> > > >
> > > > Now, the solution you have in mind will be helping with a typical
> > cluster
> > > > topology like we are usually setting up with head_node, and other flat
> > > > things
> > > > of this nature, right? In other words - a simple, "traditional"
> cluster
> > > > layout? If so, may I ask a few questions:
> > > > - how that topology will be specified?
> > > > - how it is going to be compatible with what we have right now? Not
> like
> > > I
> > > > try to preserve what we have right now, as there's not much to
> preserve.
> > > > We don't really have a way to desribe a cluster's topology at the
> > > > moment anyway.
> > > > - in case we need to describe more complex topology - how would be
> done?
> > > >
> > > > I also presume, that the proposed solution will take advantage of
> > > existing
> > > > Puppet recipes, right?
> > > >
> > > > Thanks in advance for your answers!
> > > > Cos
> > > >
> > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > > Hi folks.
> > > > >
> > > > > This last year we've made a lot of progress,
> > > > > - we can now easily build bigtop,
> > > > > - and we can test a fresh build in output/ it in vagrant boxes
> locally
> > > > >
> > > > > Whats next?
> > > > >
> > > > > In my mind, there is only one last step, to make bigtop a consumer
> > > facing
> > > > > product: We need to make it so you can simply provision a cluster in
> > > the
> > > > > cloud.
> > > > > As you may recall, David (student at RPI, intern at analytics
> company
> > > > last
> > > > > summer), asked how to do this, and spent about three weeks on it.
> > > > Recently
> > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
> University,
> > > and
> > > > > former Yahoo researchers, also asked me the same thing.
> > > > >
> > > > > Alternative?
> > > > >
> > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so
> > > on,
> > > > > which mostly are an untested, human readable version of our vagrant
> > > > > recipes. yikes !
> > > > >
> > > > > How to implement a ssh cloud bigtop installation ?
> > > > >
> > > > > We could roll our own deployer, but in doing so, we would have to
> make
> > > > > semantics for:
> > > > >
> > > > > - possible need for machiene reboots
> > > > > - syncing local folders to remote folders
> > > > > - installing (and reinstalling)
> > > > >
> > > > > luckily, our buddy Vagrant already does this for us (respectively ,
> > > with
> > > > > the "reload","synced.folder", and "up" options).
> > > > >
> > > > > Additionally, *** I THINK *** we can literally use the exact same
> > > vagrant
> > > > > recipes which we are already using to test --- so we will have a
> great
> > > > user
> > > > > experience, and really easy to reproduce bugs and test deployments.
> > > > >
> > > > >
> > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > > > suggestions
> > > > > on the implementation or feature? I think it will be easy, but also,
> > it
> > > > > will be one of the most powerful things we add to bigtop, basically
> > > > > allowing users to easily use the system... maybe even too easy :)
> > > > >
> > > > > --
> > > > > jay vyas
> > > >
> > >
> >
> >
> >
> > --
> > jay vyas
Re: polishing off bigtop : SSH Provisioning to the cloud
Posted by Konstantin Boudnik <co...@apache.org>.
Guys, just want to remind everyone that we have the DTK demo tomorrow 10 PST
- should be great!
Cos
On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote:
> We set up a goto meeting for 10:00AM PST on Thursday March 5th
>
> *Bigtop - DevOps Toolkit (DTK) Demo*
> Thu, Mar 5, 10:00 AM Pacific Standard Time
>
>
> - Please join my meeting from your computer, tablet or smartphone.
> https://global.gotomeeting.com/join/655931173
> - You can also dial in using your phone.
> United States (Long distance): +1 (571) 317-3122
> *Access Code:* 655-931-173
> More phone numbers:
> https://global.gotomeeting.com/655931173/numbersdisplay.html
>
> If interested, but this day or time does not work, please send a note and
> we can set up additional day/time(s) too.
>
> We will have a few high level slides and then have a demo that shows how we
> currently use Bigtop and how this can be naturally extended .to provide a
> "one click" solution to deploy a rich variety of bigtop stacks with
> different mixtures of services and different topologies that plugs on top
> of the existing bigtop deployment work
>
> As far as a screencast, we wanted our first one to be a short focused
> screencast; so that will be a follow up that we will put together shortly
> - Rich
>
>
>
>
> ----- Original Message -----
> From:
> dev@bigtop.apache.org
>
> To:
> <de...@bigtop.apache.org>
> Cc:
> <us...@bigtop.apache.org>
> Sent:
> Sat, 28 Feb 2015 01:31:10 +0000
> Subject:
> Re: polishing off bigtop : SSH Provisioning to the cloud
>
>
> Either of the days is open for me: would love to have a demo. Perhaps it can
> be recorded to be used as a screencast later on? ;)
>
> It'd be great to have as many people as possible on that, so we can quickly
> converge on the directions here. Adding user@ list as well...
>
> Cos
>
> On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> > Thanks Jay,
> > A screencast is something we have had on our todo list; we can easily
> > prioritize a bigtop example as the first of a few examples we were
> planning
> > to do.
> >
> > As far as the demo, some dates we had in mind were Wednesday or Thursday
> > next week.
> > I guess the easiest way to see who is interested is a reply/+1
> > with Wednesday or Thursday; we then can find exact times.
> > We are happy to schedule multiple demos at a few different days/times.
> > -Rich
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> > "dev@bigtop.apache.org" <de...@bigtop.apache.org>
> > Cc:
> >
> > Sent:
> > Wed, 25 Feb 2015 12:32:59 -0500
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud
> >
> >
> > Hi richard, also maybe even a youtube video of Devops TK deploying BigTop
> > would be awesome !
> >
> > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com>
> wrote:
> >
> > > I think might have a solution (dtk.io) that we have worked on that can
> > > help
> > > solve these problems. We have been working the last four years on the
> > > problem of deploying complex clusters. It provides a simple DSL for
> > > describing a rich set of topologies (eg simple master slave topologies,
> HA
> > > variants, security variants, monitoring and centralized logging
> > > extensions). It provides one click deployment from the selected cluster
> > > topology. It also provides a simple workflow DSL to enable finer grain
> > > control on order of execution.
> > >
> > > We are in the process now of refining a plan to open source it
> > >
> > > Our system was built to work on top of user's existing configuration
> > > assets. The initial focus has been on Puppet, but we are expanding now
> for
> > > docker support and using arbitrary scripts
> > >
> > > For last two years we have been using some of the Bigtop puppet modules
> > for
> > > our big data deployments as well as in concert with modules such as
> kafka,
> > > storm, accumulo , opentsdb ,.. We are in process now of converting to
> use
> > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
> > > leverage the groovy and gradle test work so one can bring up cluster
> stage
> > > by stage with smoke tests after every stage
> > >
> > > Think it would be straight forward so this can also plug on top of
> Bigtop
> > > vagrant work; we have a plugable iaas architecture; we currently support
> > > ec2 and "managed servers" (we have focused here on bare metal); so
> vagrant
> > > would extend reach to virtual box and its other iaas providers.
> > >
> > > I think there is potential for great synergy; best way to show would be
> > > through
> > > goto meeting demo(s) and answering any detailed questions. We can do
> this
> > > next week or the following at one or a few sessions. Will shortly send
> > > proposed dates/times
> > > - Rich
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From:
> > > dev@bigtop.apache.org
> > >
> > > To:
> > > <de...@bigtop.apache.org>
> > > Cc:
> > >
> > > Sent:
> > > Wed, 25 Feb 2015 17:24:41 +0800
> > > Subject:
> > > Re: polishing off bigtop : SSH Provisioning to the cloud.
> > >
> > >
> > > Cluster deployment and management are important things to make our work
> > > finally adopted by users. From my point of view I think there are two
> > > things we can improve further:
> > > * user friendly cluster auto deployment
> > > (we have puppet recipes, but users still need to manually ship puppet
> > > code and configurations)
> > > * topology layout
> > > (for example, specify zookeeper to be deployed on node 3~5)
> > >
> > > BIGTOP-1702 has a great potential to make cluster deployment more user
> > > friendly.
> > > That is users don't need to deal with those trivial things like ship
> > bigtop
> > > puppet code and hiera configurations or install those prerequisites all
> > > over the cluster.
> > >
> > > To answer cos's question in BIGTOP-1702:
> > >
> > > although I am not sure what you mean by
> > > link those managed servers for further provisioning
> > >
> > > According to the readme of vagrant-managed-servers, vagrant up and
> destroy
> > > are re-interpreted as "linking//unlinking". After those servers are
> > > linked, theoretically, we can directly leverage our Vagrantfile
> > > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
> > > So, this definitely will take advantage of existing puppet recipes.
> > >
> > > And here's my thought on following questions:
> > >
> > > - how that topology will be specified?
> > > - how it is going to be compatible with what we have right now? Not
> like I
> > > try to preserve what we have right now, as there's not much to preserve.
> > > We don't really have a way to desribe a cluster's topology at the
> > > moment anyway.
> > > - in case we need to describe more complex topology - how would be done?
> > >
> > > I don't think BIGTOP-1702 support topology settings. Regarding to the
> > > feature, I think we should support topology specification in our puppet
> > > recipes. We currently do not support this yet, but I believe Michael is
> > > heading toward this way.
> > >
> > >
> > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > I think it is a great idea!
> > > >
> > > > And, indeed, it's just a very small step - make it user-friendly ;)
> > > >
> > > > I believe what you're proposing below is a much better way to achieve
> > > what
> > > > I've been trying by wrapping Puppet and content distribution into
> Gradle
> > > > (you
> > > > referred to it 'our own deployer' below). That was a bad idea, in
> > > > retrospect.
> > > > And for sure - we need the functionality of this kind: we need
> something
> > > > that
> > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly
> > without
> > > > additional troubles!
> > > >
> > > > I am also a big fan of getting rid of lengthy and at times cryptic
> pages
> > > > explaining the installation process.
> > > >
> > > > Now, the solution you have in mind will be helping with a typical
> > cluster
> > > > topology like we are usually setting up with head_node, and other flat
> > > > things
> > > > of this nature, right? In other words - a simple, "traditional"
> cluster
> > > > layout? If so, may I ask a few questions:
> > > > - how that topology will be specified?
> > > > - how it is going to be compatible with what we have right now? Not
> like
> > > I
> > > > try to preserve what we have right now, as there's not much to
> preserve.
> > > > We don't really have a way to desribe a cluster's topology at the
> > > > moment anyway.
> > > > - in case we need to describe more complex topology - how would be
> done?
> > > >
> > > > I also presume, that the proposed solution will take advantage of
> > > existing
> > > > Puppet recipes, right?
> > > >
> > > > Thanks in advance for your answers!
> > > > Cos
> > > >
> > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > > Hi folks.
> > > > >
> > > > > This last year we've made a lot of progress,
> > > > > - we can now easily build bigtop,
> > > > > - and we can test a fresh build in output/ it in vagrant boxes
> locally
> > > > >
> > > > > Whats next?
> > > > >
> > > > > In my mind, there is only one last step, to make bigtop a consumer
> > > facing
> > > > > product: We need to make it so you can simply provision a cluster in
> > > the
> > > > > cloud.
> > > > > As you may recall, David (student at RPI, intern at analytics
> company
> > > > last
> > > > > summer), asked how to do this, and spent about three weeks on it.
> > > > Recently
> > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
> University,
> > > and
> > > > > former Yahoo researchers, also asked me the same thing.
> > > > >
> > > > > Alternative?
> > > > >
> > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so
> > > on,
> > > > > which mostly are an untested, human readable version of our vagrant
> > > > > recipes. yikes !
> > > > >
> > > > > How to implement a ssh cloud bigtop installation ?
> > > > >
> > > > > We could roll our own deployer, but in doing so, we would have to
> make
> > > > > semantics for:
> > > > >
> > > > > - possible need for machiene reboots
> > > > > - syncing local folders to remote folders
> > > > > - installing (and reinstalling)
> > > > >
> > > > > luckily, our buddy Vagrant already does this for us (respectively ,
> > > with
> > > > > the "reload","synced.folder", and "up" options).
> > > > >
> > > > > Additionally, *** I THINK *** we can literally use the exact same
> > > vagrant
> > > > > recipes which we are already using to test --- so we will have a
> great
> > > > user
> > > > > experience, and really easy to reproduce bugs and test deployments.
> > > > >
> > > > >
> > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > > > suggestions
> > > > > on the implementation or feature? I think it will be easy, but also,
> > it
> > > > > will be one of the most powerful things we add to bigtop, basically
> > > > > allowing users to easily use the system... maybe even too easy :)
> > > > >
> > > > > --
> > > > > jay vyas
> > > >
> > >
> >
> >
> >
> > --
> > jay vyas
Re: polishing off bigtop : SSH Provisioning to the cloud
Posted by Konstantin Boudnik <co...@apache.org>.
Count me in.
Thanks!
Cos
On Mon, Mar 02, 2015 at 12:22PM, Richard Pelavin wrote:
> We set up a goto meeting for 10:00AM PST on Thursday March 5th
>
> *Bigtop - DevOps Toolkit (DTK) Demo*
> Thu, Mar 5, 10:00 AM Pacific Standard Time
>
>
> - Please join my meeting from your computer, tablet or smartphone.
> https://global.gotomeeting.com/join/655931173
> - You can also dial in using your phone.
> United States (Long distance): +1 (571) 317-3122
> *Access Code:* 655-931-173
> More phone numbers:
> https://global.gotomeeting.com/655931173/numbersdisplay.html
>
> If interested, but this day or time does not work, please send a note and
> we can set up additional day/time(s) too.
>
> We will have a few high level slides and then have a demo that shows how we
> currently use Bigtop and how this can be naturally extended .to provide a
> "one click" solution to deploy a rich variety of bigtop stacks with
> different mixtures of services and different topologies that plugs on top
> of the existing bigtop deployment work
>
> As far as a screencast, we wanted our first one to be a short focused
> screencast; so that will be a follow up that we will put together shortly
> - Rich
>
>
>
>
> ----- Original Message -----
> From:
> dev@bigtop.apache.org
>
> To:
> <de...@bigtop.apache.org>
> Cc:
> <us...@bigtop.apache.org>
> Sent:
> Sat, 28 Feb 2015 01:31:10 +0000
> Subject:
> Re: polishing off bigtop : SSH Provisioning to the cloud
>
>
> Either of the days is open for me: would love to have a demo. Perhaps it can
> be recorded to be used as a screencast later on? ;)
>
> It'd be great to have as many people as possible on that, so we can quickly
> converge on the directions here. Adding user@ list as well...
>
> Cos
>
> On Wed, Feb 25, 2015 at 04:09PM, Richard Pelavin wrote:
> > Thanks Jay,
> > A screencast is something we have had on our todo list; we can easily
> > prioritize a bigtop example as the first of a few examples we were
> planning
> > to do.
> >
> > As far as the demo, some dates we had in mind were Wednesday or Thursday
> > next week.
> > I guess the easiest way to see who is interested is a reply/+1
> > with Wednesday or Thursday; we then can find exact times.
> > We are happy to schedule multiple demos at a few different days/times.
> > -Rich
> >
> >
> > ----- Original Message -----
> > From:
> > dev@bigtop.apache.org
> >
> > To:
> > "dev@bigtop.apache.org" <de...@bigtop.apache.org>
> > Cc:
> >
> > Sent:
> > Wed, 25 Feb 2015 12:32:59 -0500
> > Subject:
> > Re: polishing off bigtop : SSH Provisioning to the cloud
> >
> >
> > Hi richard, also maybe even a youtube video of Devops TK deploying BigTop
> > would be awesome !
> >
> > On Wed, Feb 25, 2015 at 12:00 PM, Richard Pelavin <ri...@reactor8.com>
> wrote:
> >
> > > I think might have a solution (dtk.io) that we have worked on that can
> > > help
> > > solve these problems. We have been working the last four years on the
> > > problem of deploying complex clusters. It provides a simple DSL for
> > > describing a rich set of topologies (eg simple master slave topologies,
> HA
> > > variants, security variants, monitoring and centralized logging
> > > extensions). It provides one click deployment from the selected cluster
> > > topology. It also provides a simple workflow DSL to enable finer grain
> > > control on order of execution.
> > >
> > > We are in the process now of refining a plan to open source it
> > >
> > > Our system was built to work on top of user's existing configuration
> > > assets. The initial focus has been on Puppet, but we are expanding now
> for
> > > docker support and using arbitrary scripts
> > >
> > > For last two years we have been using some of the Bigtop puppet modules
> > for
> > > our big data deployments as well as in concert with modules such as
> kafka,
> > > storm, accumulo , opentsdb ,.. We are in process now of converting to
> use
> > > the new Bigtop Puppet 3.x hiera modules. We also just put in ability to
> > > leverage the groovy and gradle test work so one can bring up cluster
> stage
> > > by stage with smoke tests after every stage
> > >
> > > Think it would be straight forward so this can also plug on top of
> Bigtop
> > > vagrant work; we have a plugable iaas architecture; we currently support
> > > ec2 and "managed servers" (we have focused here on bare metal); so
> vagrant
> > > would extend reach to virtual box and its other iaas providers.
> > >
> > > I think there is potential for great synergy; best way to show would be
> > > through
> > > goto meeting demo(s) and answering any detailed questions. We can do
> this
> > > next week or the following at one or a few sessions. Will shortly send
> > > proposed dates/times
> > > - Rich
> > >
> > >
> > >
> > > ----- Original Message -----
> > > From:
> > > dev@bigtop.apache.org
> > >
> > > To:
> > > <de...@bigtop.apache.org>
> > > Cc:
> > >
> > > Sent:
> > > Wed, 25 Feb 2015 17:24:41 +0800
> > > Subject:
> > > Re: polishing off bigtop : SSH Provisioning to the cloud.
> > >
> > >
> > > Cluster deployment and management are important things to make our work
> > > finally adopted by users. From my point of view I think there are two
> > > things we can improve further:
> > > * user friendly cluster auto deployment
> > > (we have puppet recipes, but users still need to manually ship puppet
> > > code and configurations)
> > > * topology layout
> > > (for example, specify zookeeper to be deployed on node 3~5)
> > >
> > > BIGTOP-1702 has a great potential to make cluster deployment more user
> > > friendly.
> > > That is users don't need to deal with those trivial things like ship
> > bigtop
> > > puppet code and hiera configurations or install those prerequisites all
> > > over the cluster.
> > >
> > > To answer cos's question in BIGTOP-1702:
> > >
> > > although I am not sure what you mean by
> > > link those managed servers for further provisioning
> > >
> > > According to the readme of vagrant-managed-servers, vagrant up and
> destroy
> > > are re-interpreted as "linking//unlinking". After those servers are
> > > linked, theoretically, we can directly leverage our Vagrantfile
> > > in "vagrant-puppet-vm" to do the whole cluster deployment by one-click.
> > > So, this definitely will take advantage of existing puppet recipes.
> > >
> > > And here's my thought on following questions:
> > >
> > > - how that topology will be specified?
> > > - how it is going to be compatible with what we have right now? Not
> like I
> > > try to preserve what we have right now, as there's not much to preserve.
> > > We don't really have a way to desribe a cluster's topology at the
> > > moment anyway.
> > > - in case we need to describe more complex topology - how would be done?
> > >
> > > I don't think BIGTOP-1702 support topology settings. Regarding to the
> > > feature, I think we should support topology specification in our puppet
> > > recipes. We currently do not support this yet, but I believe Michael is
> > > heading toward this way.
> > >
> > >
> > > 2015-02-25 15:33 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > > > I think it is a great idea!
> > > >
> > > > And, indeed, it's just a very small step - make it user-friendly ;)
> > > >
> > > > I believe what you're proposing below is a much better way to achieve
> > > what
> > > > I've been trying by wrapping Puppet and content distribution into
> Gradle
> > > > (you
> > > > referred to it 'our own deployer' below). That was a bad idea, in
> > > > retrospect.
> > > > And for sure - we need the functionality of this kind: we need
> something
> > > > that
> > > > let anyone to deploy a cluster to a pre-provisioned nodes quickly
> > without
> > > > additional troubles!
> > > >
> > > > I am also a big fan of getting rid of lengthy and at times cryptic
> pages
> > > > explaining the installation process.
> > > >
> > > > Now, the solution you have in mind will be helping with a typical
> > cluster
> > > > topology like we are usually setting up with head_node, and other flat
> > > > things
> > > > of this nature, right? In other words - a simple, "traditional"
> cluster
> > > > layout? If so, may I ask a few questions:
> > > > - how that topology will be specified?
> > > > - how it is going to be compatible with what we have right now? Not
> like
> > > I
> > > > try to preserve what we have right now, as there's not much to
> preserve.
> > > > We don't really have a way to desribe a cluster's topology at the
> > > > moment anyway.
> > > > - in case we need to describe more complex topology - how would be
> done?
> > > >
> > > > I also presume, that the proposed solution will take advantage of
> > > existing
> > > > Puppet recipes, right?
> > > >
> > > > Thanks in advance for your answers!
> > > > Cos
> > > >
> > > > On Tue, Feb 24, 2015 at 09:32AM, jay vyas wrote:
> > > > > Hi folks.
> > > > >
> > > > > This last year we've made a lot of progress,
> > > > > - we can now easily build bigtop,
> > > > > - and we can test a fresh build in output/ it in vagrant boxes
> locally
> > > > >
> > > > > Whats next?
> > > > >
> > > > > In my mind, there is only one last step, to make bigtop a consumer
> > > facing
> > > > > product: We need to make it so you can simply provision a cluster in
> > > the
> > > > > cloud.
> > > > > As you may recall, David (student at RPI, intern at analytics
> company
> > > > last
> > > > > summer), asked how to do this, and spent about three weeks on it.
> > > > Recently
> > > > > Ata Turk, working on the Massachusets Open Cloud @ Boston
> University,
> > > and
> > > > > former Yahoo researchers, also asked me the same thing.
> > > > >
> > > > > Alternative?
> > > > >
> > > > > Well we currently these docs about how to setup 0.7.0, 0.8.0, and so
> > > on,
> > > > > which mostly are an untested, human readable version of our vagrant
> > > > > recipes. yikes !
> > > > >
> > > > > How to implement a ssh cloud bigtop installation ?
> > > > >
> > > > > We could roll our own deployer, but in doing so, we would have to
> make
> > > > > semantics for:
> > > > >
> > > > > - possible need for machiene reboots
> > > > > - syncing local folders to remote folders
> > > > > - installing (and reinstalling)
> > > > >
> > > > > luckily, our buddy Vagrant already does this for us (respectively ,
> > > with
> > > > > the "reload","synced.folder", and "up" options).
> > > > >
> > > > > Additionally, *** I THINK *** we can literally use the exact same
> > > vagrant
> > > > > recipes which we are already using to test --- so we will have a
> great
> > > > user
> > > > > experience, and really easy to reproduce bugs and test deployments.
> > > > >
> > > > >
> > > > > I';l hash out details in BIGTOP-1702 , any thoughts, questions,
> > > > suggestions
> > > > > on the implementation or feature? I think it will be easy, but also,
> > it
> > > > > will be one of the most powerful things we add to bigtop, basically
> > > > > allowing users to easily use the system... maybe even too easy :)
> > > > >
> > > > > --
> > > > > jay vyas
> > > >
> > >
> >
> >
> >
> > --
> > jay vyas