Wrapping more functionality into Gradle

[Konstantin Boudnik <co...@apache.org>]

Guys,

I've been doing some thinking lately (yes, I know I shouldn't - I've heard
this from good friends before, but can't help it) and it makes more and more
sense to me to wrap more build/deploy functionality into our Gradle build.

The low-hanging fruits I have in mind right now is to provide a wrapper for
{{puppet apply bigtop-deploy/puppet/manifests/site.pp}} that will do a few
things like:
 - copy deployment recipes to all specified hosts, including site.csv file
 - start puppet apply on all hosts in parallel using gradle ssh plugin

This, IMO, will provide us with a very easy and transparent way of quickly
deploying cluster bits as one works on something, without switching out of the
convenient build environment. Sorta, change-deploy-rinse-and-repeat loop.

The same gradle task can be called, if desired from Jenkins or else.

Do you think it worth it? Thoughts?
-- 
Take care,
  Cos

Re: Wrapping more functionality into Gradle

[Konstantin Boudnik <co...@apache.org>]

On Thu, Dec 25, 2014 at 01:26PM, Evans Ye wrote:
> If the idea is to propose a functional README,
> I think this will do a lot of help for new comers to get things work
> functionally.
> I myself once staring on the readme figuring out how to "apply".
> Though the gradle things may not interest ops people, it will make Bigtop
> modules much more friendly for users.

Indeed. Hopefully, I will have something before the week end.
  Cos

Re: Wrapping more functionality into Gradle

[Jay Vyas <ja...@gmail.com>]

I think the trending idea is :

Gradle is a user interface to BigTop.

In that spirit I think we can also more generally start thinking about how to use and architect the gradle tasks as a user interface.

> On Dec 24, 2014, at 11:26 PM, Evans Ye <sa...@gmail.com> wrote:
> 
> If the idea is to propose a functional README,
> I think this will do a lot of help for new comers to get things work
> functionally.
> I myself once staring on the readme figuring out how to "apply".
> Though the gradle things may not interest ops people, it will make Bigtop
> modules much more friendly for users.

Re: Wrapping more functionality into Gradle

[Evans Ye <sa...@gmail.com>]

If the idea is to propose a functional README,
I think this will do a lot of help for new comers to get things work
functionally.
I myself once staring on the readme figuring out how to "apply".
Though the gradle things may not interest ops people, it will make Bigtop
modules much more friendly for users.

Re: Wrapping more functionality into Gradle

[Konstantin Boudnik <co...@apache.org>]

On Tue, Dec 23, 2014 at 02:12PM, Roman Shaposhnik wrote:
> On Tue, Dec 23, 2014 at 12:14 AM, Konstantin Boudnik <co...@apache.org> wrote:
> > Yeah, I am pretty sceptical about master-based setups as well. In my
> > understanding, for proper functioning of such a setup the recipes should be
> > machine-specific or group-specific. While in our case we don't have any
> > up-front information where the recipes will be executed. Hence, our way of
> > using Puppet seems to be pretty well aligned with our use cases.
> >
> > And getting back to topic: I take it the idea of adding Gradle's end-point for
> > Puppet apply doesn't look too sexy, right?
> 
> It does to me, but really -- how much work are we talking about here? Just
> adding a task that wraps around what's in our README today?

Yes, pretty much. It's like having a functional README. Ok, lemme make a patch
for that and we'll see if there's enough interest to commit it.

Thanks,
  Cos

Re: Wrapping more functionality into Gradle

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Tue, Dec 23, 2014 at 12:14 AM, Konstantin Boudnik <co...@apache.org> wrote:
> Yeah, I am pretty sceptical about master-based setups as well. In my
> understanding, for proper functioning of such a setup the recipes should be
> machine-specific or group-specific. While in our case we don't have any
> up-front information where the recipes will be executed. Hence, our way of
> using Puppet seems to be pretty well aligned with our use cases.
>
> And getting back to topic: I take it the idea of adding Gradle's end-point for
> Puppet apply doesn't look too sexy, right?

It does to me, but really -- how much work are we talking about here? Just
adding a task that wraps around what's in our README today?

Thanks,
Roman.

Re: Wrapping more functionality into Gradle

[Konstantin Boudnik <co...@apache.org>]

Yeah, I am pretty sceptical about master-based setups as well. In my
understanding, for proper functioning of such a setup the recipes should be
machine-specific or group-specific. While in our case we don't have any
up-front information where the recipes will be executed. Hence, our way of
using Puppet seems to be pretty well aligned with our use cases.

And getting back to topic: I take it the idea of adding Gradle's end-point for
Puppet apply doesn't look too sexy, right?

Cos

On Sun, Dec 21, 2014 at 11:41PM, Evans Ye wrote:
> Regarding to the puppetmaster, I'd like to raise a question that does most
> of the company using puppet in this way?
> I have experience using puppet master for cluster management and deployment.
> But there are troubles I ran into and I think it would be better to switch
> to masterless puppet.
> When I searched around on net, I found that actually there are many
> companies already did the switch.
> A typical solution for using masterless puppet is to use version
> control(git) with puppet apply.
> (though I'm not sure whether there is a better way)
> 
> To increase the Bigtop adoption by improving our puppet stuff, I think
> there are some items can be put on for discussion:
> 
> * Journal HA for Namenodes
> * find-grained configuration on roles
> (some machines run datanode+zookeeper+journalnode, some machines only run
> nodemanager)
> 
> However, this is only my personal thoughts.
> Would love to hear yours.
> 
> Getting back to the topic, I think the gradle thing can better demonstrate
> the usage of bigtop puppet, though some of the ops people might not want
> the extra dependency:)
> 
> 
> 2014-12-20 12:17 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> 
> > Thanks for kind words, man - I am glad to see that Gradle took off!
> >
> > Let me clarify something - sorry for not being clear right away. The
> > application of the wrapper isn't to manage LIVE clusters or machines. Also,
> > Puppet recipes aren't going away neither. All I had in mind is to provide
> > an
> > easy wrapper, so that
> >
> > a) don't bother with copying around deployment script when you need to
> > quickly
> >    get a dev. cluster up and running
> > b) one doesn't need to type the long command wrapped in pdsh or otherwise
> >
> > It might be or might not be a good idea and that's why I wanted to get it
> > out
> > there for everyone to bash on ;)
> >
> > Cos
> >
> > On Fri, Dec 19, 2014 at 09:25PM, jay vyas wrote:
> > > Hi cos!  I like the goal, but let me pervert your idea a little bit :)
> > ....
> > > how does this sound ?
> > >
> > > *** Definetely ***
> > >
> > > lets keep going w/ gradle for all build/test/deploy stuff.  Its great for
> > > that !  Full court press to remove maven deps, gradle is wayyyy cleaner
> > and
> > > definetely is the future trend.  the hard work you've done in integrating
> > > the whole build to use it is really making bigtop a fun to use packaging
> > > system, and actually has made me more able to contribute to it b/c its
> > > easier to debug the build now for guys like me that dont know much about
> > > make.
> > >
> > > *** BUT ***
> > >
> > > although I LOVE the idea of adding a CONTROLLER to the live machines, ...
> > > adding it in gradle? very few devops folks will really want to use gradle
> > > for that kind of task....  its not the idiom.
> > >
> > > One of the HUGE selling points of bigtop is that it has puppet recipes,
> > > rather than just one of commands/tasks for managing the cluster - the ops
> > > community can read and understand those easily.
> > >
> > > *** SO : PUPPET MASTER .... AN ALTERNATIVE ? ***
> > >
> > > I think, if we want to manage LIVE machines, then can i suggest a real
> > > puppet master implementation? If we go that route, you get all the
> > testing
> > > for it also, for free, w/ our existing vagrant recipes.... and its in
> > line
> > > w/ what the devops community is doing, and so it will help drive bigtop
> > > adoption.
> > >
> > >
> > > On Fri, Dec 19, 2014 at 8:19 PM, Konstantin Boudnik <co...@apache.org>
> > wrote:
> > > >
> > > > Guys,
> > > >
> > > > I've been doing some thinking lately (yes, I know I shouldn't - I've
> > heard
> > > > this from good friends before, but can't help it) and it makes more and
> > > > more
> > > > sense to me to wrap more build/deploy functionality into our Gradle
> > build.
> > > >
> > > > The low-hanging fruits I have in mind right now is to provide a
> > wrapper for
> > > > {{puppet apply bigtop-deploy/puppet/manifests/site.pp}} that will do a
> > few
> > > > things like:
> > > >  - copy deployment recipes to all specified hosts, including site.csv
> > file
> > > >  - start puppet apply on all hosts in parallel using gradle ssh plugin
> > > >
> > > > This, IMO, will provide us with a very easy and transparent way of
> > quickly
> > > > deploying cluster bits as one works on something, without switching
> > out of
> > > > the
> > > > convenient build environment. Sorta, change-deploy-rinse-and-repeat
> > loop.
> > > >
> > > > The same gradle task can be called, if desired from Jenkins or else.
> > > >
> > > > Do you think it worth it? Thoughts?
> > > > --
> > > > Take care,
> > > >   Cos
> > > >
> > > >
> > >
> > > --
> > > jay vyas
> >

Re: Wrapping more functionality into Gradle

[Evans Ye <sa...@gmail.com>]

Regarding to the puppetmaster, I'd like to raise a question that does most
of the company using puppet in this way?
I have experience using puppet master for cluster management and deployment.
But there are troubles I ran into and I think it would be better to switch
to masterless puppet.
When I searched around on net, I found that actually there are many
companies already did the switch.
A typical solution for using masterless puppet is to use version
control(git) with puppet apply.
(though I'm not sure whether there is a better way)

To increase the Bigtop adoption by improving our puppet stuff, I think
there are some items can be put on for discussion:

* Journal HA for Namenodes
* find-grained configuration on roles
(some machines run datanode+zookeeper+journalnode, some machines only run
nodemanager)

However, this is only my personal thoughts.
Would love to hear yours.

Getting back to the topic, I think the gradle thing can better demonstrate
the usage of bigtop puppet, though some of the ops people might not want
the extra dependency:)


2014-12-20 12:17 GMT+08:00 Konstantin Boudnik <co...@apache.org>:

> Thanks for kind words, man - I am glad to see that Gradle took off!
>
> Let me clarify something - sorry for not being clear right away. The
> application of the wrapper isn't to manage LIVE clusters or machines. Also,
> Puppet recipes aren't going away neither. All I had in mind is to provide
> an
> easy wrapper, so that
>
> a) don't bother with copying around deployment script when you need to
> quickly
>    get a dev. cluster up and running
> b) one doesn't need to type the long command wrapped in pdsh or otherwise
>
> It might be or might not be a good idea and that's why I wanted to get it
> out
> there for everyone to bash on ;)
>
> Cos
>
> On Fri, Dec 19, 2014 at 09:25PM, jay vyas wrote:
> > Hi cos!  I like the goal, but let me pervert your idea a little bit :)
> ....
> > how does this sound ?
> >
> > *** Definetely ***
> >
> > lets keep going w/ gradle for all build/test/deploy stuff.  Its great for
> > that !  Full court press to remove maven deps, gradle is wayyyy cleaner
> and
> > definetely is the future trend.  the hard work you've done in integrating
> > the whole build to use it is really making bigtop a fun to use packaging
> > system, and actually has made me more able to contribute to it b/c its
> > easier to debug the build now for guys like me that dont know much about
> > make.
> >
> > *** BUT ***
> >
> > although I LOVE the idea of adding a CONTROLLER to the live machines, ...
> > adding it in gradle? very few devops folks will really want to use gradle
> > for that kind of task....  its not the idiom.
> >
> > One of the HUGE selling points of bigtop is that it has puppet recipes,
> > rather than just one of commands/tasks for managing the cluster - the ops
> > community can read and understand those easily.
> >
> > *** SO : PUPPET MASTER .... AN ALTERNATIVE ? ***
> >
> > I think, if we want to manage LIVE machines, then can i suggest a real
> > puppet master implementation? If we go that route, you get all the
> testing
> > for it also, for free, w/ our existing vagrant recipes.... and its in
> line
> > w/ what the devops community is doing, and so it will help drive bigtop
> > adoption.
> >
> >
> > On Fri, Dec 19, 2014 at 8:19 PM, Konstantin Boudnik <co...@apache.org>
> wrote:
> > >
> > > Guys,
> > >
> > > I've been doing some thinking lately (yes, I know I shouldn't - I've
> heard
> > > this from good friends before, but can't help it) and it makes more and
> > > more
> > > sense to me to wrap more build/deploy functionality into our Gradle
> build.
> > >
> > > The low-hanging fruits I have in mind right now is to provide a
> wrapper for
> > > {{puppet apply bigtop-deploy/puppet/manifests/site.pp}} that will do a
> few
> > > things like:
> > >  - copy deployment recipes to all specified hosts, including site.csv
> file
> > >  - start puppet apply on all hosts in parallel using gradle ssh plugin
> > >
> > > This, IMO, will provide us with a very easy and transparent way of
> quickly
> > > deploying cluster bits as one works on something, without switching
> out of
> > > the
> > > convenient build environment. Sorta, change-deploy-rinse-and-repeat
> loop.
> > >
> > > The same gradle task can be called, if desired from Jenkins or else.
> > >
> > > Do you think it worth it? Thoughts?
> > > --
> > > Take care,
> > >   Cos
> > >
> > >
> >
> > --
> > jay vyas
>

Re: Wrapping more functionality into Gradle

[Konstantin Boudnik <co...@apache.org>]

Thanks for kind words, man - I am glad to see that Gradle took off!

Let me clarify something - sorry for not being clear right away. The
application of the wrapper isn't to manage LIVE clusters or machines. Also,
Puppet recipes aren't going away neither. All I had in mind is to provide an
easy wrapper, so that 

a) don't bother with copying around deployment script when you need to quickly
   get a dev. cluster up and running
b) one doesn't need to type the long command wrapped in pdsh or otherwise

It might be or might not be a good idea and that's why I wanted to get it out
there for everyone to bash on ;)

Cos

On Fri, Dec 19, 2014 at 09:25PM, jay vyas wrote:
> Hi cos!  I like the goal, but let me pervert your idea a little bit :) ....
> how does this sound ?
> 
> *** Definetely ***
> 
> lets keep going w/ gradle for all build/test/deploy stuff.  Its great for
> that !  Full court press to remove maven deps, gradle is wayyyy cleaner and
> definetely is the future trend.  the hard work you've done in integrating
> the whole build to use it is really making bigtop a fun to use packaging
> system, and actually has made me more able to contribute to it b/c its
> easier to debug the build now for guys like me that dont know much about
> make.
> 
> *** BUT ***
> 
> although I LOVE the idea of adding a CONTROLLER to the live machines, ...
> adding it in gradle? very few devops folks will really want to use gradle
> for that kind of task....  its not the idiom.
> 
> One of the HUGE selling points of bigtop is that it has puppet recipes,
> rather than just one of commands/tasks for managing the cluster - the ops
> community can read and understand those easily.
> 
> *** SO : PUPPET MASTER .... AN ALTERNATIVE ? ***
> 
> I think, if we want to manage LIVE machines, then can i suggest a real
> puppet master implementation? If we go that route, you get all the testing
> for it also, for free, w/ our existing vagrant recipes.... and its in line
> w/ what the devops community is doing, and so it will help drive bigtop
> adoption.
> 
> 
> On Fri, Dec 19, 2014 at 8:19 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >
> > Guys,
> >
> > I've been doing some thinking lately (yes, I know I shouldn't - I've heard
> > this from good friends before, but can't help it) and it makes more and
> > more
> > sense to me to wrap more build/deploy functionality into our Gradle build.
> >
> > The low-hanging fruits I have in mind right now is to provide a wrapper for
> > {{puppet apply bigtop-deploy/puppet/manifests/site.pp}} that will do a few
> > things like:
> >  - copy deployment recipes to all specified hosts, including site.csv file
> >  - start puppet apply on all hosts in parallel using gradle ssh plugin
> >
> > This, IMO, will provide us with a very easy and transparent way of quickly
> > deploying cluster bits as one works on something, without switching out of
> > the
> > convenient build environment. Sorta, change-deploy-rinse-and-repeat loop.
> >
> > The same gradle task can be called, if desired from Jenkins or else.
> >
> > Do you think it worth it? Thoughts?
> > --
> > Take care,
> >   Cos
> >
> >
> 
> -- 
> jay vyas

Re: Wrapping more functionality into Gradle

[jay vyas <ja...@gmail.com>]

Hi cos!  I like the goal, but let me pervert your idea a little bit :) ....
how does this sound ?

*** Definetely ***

lets keep going w/ gradle for all build/test/deploy stuff.  Its great for
that !  Full court press to remove maven deps, gradle is wayyyy cleaner and
definetely is the future trend.  the hard work you've done in integrating
the whole build to use it is really making bigtop a fun to use packaging
system, and actually has made me more able to contribute to it b/c its
easier to debug the build now for guys like me that dont know much about
make.

*** BUT ***

although I LOVE the idea of adding a CONTROLLER to the live machines, ...
adding it in gradle? very few devops folks will really want to use gradle
for that kind of task....  its not the idiom.

One of the HUGE selling points of bigtop is that it has puppet recipes,
rather than just one of commands/tasks for managing the cluster - the ops
community can read and understand those easily.

*** SO : PUPPET MASTER .... AN ALTERNATIVE ? ***

I think, if we want to manage LIVE machines, then can i suggest a real
puppet master implementation? If we go that route, you get all the testing
for it also, for free, w/ our existing vagrant recipes.... and its in line
w/ what the devops community is doing, and so it will help drive bigtop
adoption.


On Fri, Dec 19, 2014 at 8:19 PM, Konstantin Boudnik <co...@apache.org> wrote:
>
> Guys,
>
> I've been doing some thinking lately (yes, I know I shouldn't - I've heard
> this from good friends before, but can't help it) and it makes more and
> more
> sense to me to wrap more build/deploy functionality into our Gradle build.
>
> The low-hanging fruits I have in mind right now is to provide a wrapper for
> {{puppet apply bigtop-deploy/puppet/manifests/site.pp}} that will do a few
> things like:
>  - copy deployment recipes to all specified hosts, including site.csv file
>  - start puppet apply on all hosts in parallel using gradle ssh plugin
>
> This, IMO, will provide us with a very easy and transparent way of quickly
> deploying cluster bits as one works on something, without switching out of
> the
> convenient build environment. Sorta, change-deploy-rinse-and-repeat loop.
>
> The same gradle task can be called, if desired from Jenkins or else.
>
> Do you think it worth it? Thoughts?
> --
> Take care,
>   Cos
>
>

-- 
jay vyas

Re: Wrapping more functionality into Gradle

[Roman Shaposhnik <ro...@shaposhnik.org>]

On Fri, Dec 19, 2014 at 5:19 PM, Konstantin Boudnik <co...@apache.org> wrote:
> This, IMO, will provide us with a very easy and transparent way of quickly
> deploying cluster bits as one works on something, without switching out of the
> convenient build environment. Sorta, change-deploy-rinse-and-repeat loop.
>
> The same gradle task can be called, if desired from Jenkins or else.
>
> Do you think it worth it? Thoughts?

Personally, I think that using Gradle as a sort of repository (or entry point
if you will) for everything that we do in Bigtop would be super useful. The
alternative is really just sh/bash and Gradle offers a much nicer DSL

Thanks,
Roman.