You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Philip Herron <ph...@wandisco.com> on 2013/03/27 14:54:23 UTC

Python Fork

Hey all

I am new to bigtop and I would like to get feedback on this initial
concept from the community on this.

I have created a fork of bigtop in my own time at:

http://git.buildy.org/?p=bigtop.git;a=shortlog;h=refs/heads/python-fork

  $ git clone git://buildy.org/bigtop.git
  $ git checkout --track -b python-fork origin/python-fork

This fork of bigtop's toplevel Make/Bash logic within python has a
number of benefits to the whole community of bigtop for its most
important function namely packaging.

1 - Bigtop's logic is handled within gnu/make and bash which is
detrimental for the long term maintainability of bigtop as it may turn
away developers.

2 - Make is hard to debug in this way esp for the need of \ for line run-on.

3 - This new directory structure feels more maintainable already with
bigtop's main function to generate packages for hadoop stack's on
different platforms. Other functions which (don't seem) to be actively
maintained like bigtop-deploy and the test framework with regards to
their documentation.

4 - Using python we can actually take advantage of the system. Using
threads to do multiple builds of different components at the same time.
Yes i know we could probably hack together make -j n but this will
further make things harder to maintain.

5 - The bigtop.mk BOM file was difficult to read and find errors with
eval and ?= as well as having archives set in Makefile and package.mk
doing a lot of tricky bash to get things going.

Using Python ConfigParser we can have a single BOM file for all the
dynamic data we care about instead of over several makefiles.

### bigtop.BOM ###
[DEFAULT]
APACHE_MIRROR = http://apache.osuosl.org
APACHE_ARCHIVE = http://archive.apache.org/dist
ARCHIVES = %(APACHE_MIRROR)s %(APACHE_ARCHIVE)s

[BIGTOP]
BOM = hadoop

[hadoop]
NAME = hadoop
BASE_VERSION = 2.0.3-alpha
PKG_VERSION = 2.0.3
RELEASE = 1
SRC_TYPE = TARGZ
SRC = %(NAME)s-%(BASE_VERSION)s-src.tar.gz
DST = %(NAME)s-%(BASE_VERSION)s.tar.gz
LOC = %(ARCHIVES)s
DOWNLOAD_PATH = /hadoop/core/%(NAME)s-%(BASE_VERSION)s
### EOF ###

I need to spend time documenting but i think this looks very similar but
cleaner. We have DEFAULTS section to specify our archives then a BOM
section listing all the components we want in the stack.

The Hadoop component is the only one i have tested with but i don’t see
any problems adding others. The LOC field in the hadoop component is
interesting as we can specify the ARCHIVES and it will look in all for
the tarball we want if the first was down it will go and look into the
next one if you have several specified.

----

Over all i could go on and talk about more issues and things that need
looked at and why this is a good step for bigtop to look at taking but i
think feedback from the community is more important now.

One thing to note is this isn't any extra dependency as this is just
stock python code no need for pip install or easy_install of modules.
And almost all distro's of linux/unix come with python so i think its fine.

Thanks for reading this! I will aim to complete the BOM against trunk
bigtop and see if i come against anything.

--Phil

Re: Python Fork

Posted by Bruno Mahé <bm...@apache.org>.
On 03/28/2013 08:04 AM, Philip Herron wrote:
> On 27/03/13 15:37, Roman Shaposhnik wrote:
>> Hi Philip,
>>
>> first of all thanks a bunch for considering to lead with the code -- that's
>> the best way to make progress. I have just started to look at your code
>> and haven finished reviewing it yet (in fact, I'd rather have somebody
>> on this list who appreciates Python doing that -- Steve?) but let me
>> quickly answer some of the points you've raised.
>>
>> My only metapoint here would be that since we're stuck with Java
>> because 90% of Bigtop components are implemented in it
>> and we've also chosen it as a platform for iTest -- any solution
>> that leverages Java tooling (Gradle, Groovy, etc) would get
>> more of my personal enthusiastic embrace than a solution
>> that introduces yet another platform to care about (Python,
>> Ruby, etc.).
>
> I understand your point here but i think python suits this project much
> better than a jvm based language because of the dependency of the jvm to
> me isn't scripting. Python gets by this because its much simpler and
> installed even in minimal environments of linux by default. And suits
> this work very well.
>
> At the moment we care about:
>
> Make/Bash/java/puppet/ruby/kickstart/ From all i can see at the moment,
> With python we can unify all of bigtop's functionality properly.
>
> With using puppet and i intend to start a fork of vmbuilder to unify all
> these projects in making images of virtual machines. Since vmbuilder is
> basically ubuntu only now with centos support floating about in github.
> Boxgrinder is kind of gross to get setup because of ruby.
>
>>
>> On Wed, Mar 27, 2013 at 6:54 AM, Philip Herron
>> <ph...@wandisco.com> wrote:
>>> 1 - Bigtop's logic is handled within gnu/make and bash which is
>>> detrimental for the long term maintainability of bigtop as it may turn
>>> away developers.
>>>
>>> 2 - Make is hard to debug in this way esp for the need of \ for line run-on.
>>
>> Could you, please, elaborate on what exact problem you were faced
>> with when you decided that Python is a better option than Make?
>> We all have preferences (for example mine is, honestly, make over
>> python) but of course if the current tooling is inadequate in some
>> respect it needs to be updated.
>
> Yeah I understand you like make i don't mind it but it doesn't have any
> real long term potential. Because i tend to think most people from java
> backgrounds will have next to no background with make. I come from a gcc
> developer/commiter background so i am used to it. But even then this
> isn't a valid use of make in my opinion and i am sure gnu/make
> developers would agree this isn't how make should be used.
>
> Take for example, the next latest and greatest component wants to be a
> *.{zip,tar.xz.tar.bz2,.xz,*.7z} this is a basic example but its still
> valid having the dynamic support for different archives how would we do
> this in python we could figure this out. In Bash were going to have to
> have more hooks and more obscure bash to do something like that.
>
>>
>>> 3 - This new directory structure feels more maintainable already with
>>> bigtop's main function to generate packages for hadoop stack's on
>>> different platforms. Other functions which (don't seem) to be actively
>>> maintained like bigtop-deploy and the test framework with regards to
>>> their documentation.
>>
>> Here's my very strong take on it -- without deployment and test Bigtop
>> might as well pack its stuff and go home. If we're reduced to a "framework
>> that builds packages" there's very little left in the project to keep me
>> interested in it.
>
> Deployment and test can still be done and there are 100% important i
> think if i come back with a more complete bigtop with these components
> my fork might be taken more seriously.
>
>>
>> We *all* should be investing pretty heavily in both. In fact, I hoped
>> to make Bigtop 0.6.0 about polishing our tests and deployment
>> code, but Hadoop 2.0.3 story changed that. Hopefully Bigtop 0.7.0
>> will be a release where we really do invest a lot of energy in
>> making both squeaky clean.
>>
>
> I understand this, it will be nice to see them.
>
>>> 5 - The bigtop.mk BOM file was difficult to read and find errors with
>>> eval and ?= as well as having archives set in Makefile and package.mk
>>> doing a lot of tricky bash to get things going.
>>
>> Could you, please elaborate? What exactly made it difficult? It is,
>> essentially, a bunch of key-value pairs (with $substitutions).
>
> Yes its key value you pairs but its horrible to read. Coupled with a
> $eval at the end of each component. I dunno this is a glorified config
> file and i tend to be very pragmatic about this and think this isn't
> what it should look like.
>
>>
>>> One thing to note is this isn't any extra dependency as this is just
>>> stock python code no need for pip install or easy_install of modules.
>>> And almost all distro's of linux/unix come with python so i think its fine.
>>
>> That's my biggest worry -- I had *terrible* luck with Python breaking
>> between different Linux distributions.
>>
>> Thanks,
>> Roman.
>>
>
> I understand your worry there, this code is pure python no extra depends
> for now. So that's good and if i get rid of my with usage i can make it
> python >= 2.4 and this is eliminate this worry dead.
>
> Java is by far worse for this problem. example:
> gcj/eclipse/openjdk/oracle...{5,6,7,8}?
>
> And how do you get a standard java version on each linux distro?
>
> I will come back with a more solid implementation with deployment and
> testing.
>
> --Phil
>


Frankly, whether it's java, python, scala or any other language (erlang 
included), they would all work for this purpose. Our build system is not 
that complicated and any of these languages would be better than our 
makefile, even if I have grown fond of it.


In this case I would tend to prefer python because it requires less 
dependencies (ie. I don't have groovy installed and I don't plan on 
installing it, and no maven/gradle would not make it any better).


Thanks,
Bruno

Re: Python Fork

Posted by Roman Shaposhnik <rv...@apache.org>.
On Thu, Mar 28, 2013 at 8:04 AM, Philip Herron
<ph...@wandisco.com> wrote:
>>> One thing to note is this isn't any extra dependency as this is just
>>> stock python code no need for pip install or easy_install of modules.
>>> And almost all distro's of linux/unix come with python so i think its fine.
>>
>> That's my biggest worry -- I had *terrible* luck with Python breaking
>> between different Linux distributions.
>
> I understand your worry there, this code is pure python no extra depends
> for now. So that's good and if i get rid of my with usage i can make it
> python >= 2.4 and this is eliminate this worry dead.
>
> Java is by far worse for this problem. example:
> gcj/eclipse/openjdk/oracle...{5,6,7,8}?

You're correct that JDK doesn't usually get deployed on a typical
Linux distro (although that is changing with OpenJDK7).

Now, I know there's little chance of me convincing you, but at
least I can related a personal experience. I've been around Java
since 1998. I've been around Python since 2002 or so. If we
pick 2002 as a cut off point I've literally upgraded between a couple
of dozen different JDKs and I didn't have a single problem
with the bytcode being accepted and correctly executed. Honestly
not even once -- may be I'm just lucky that way. With Python it
feels like every single time I upgrade *something* breaks. And I
admit a lot of times it is poorly written extensions or native extensions,
etc. But as a user of a *platform* I don't care -- it makes me
distrust the platform.

In fact, as far as platforms go (notice -- I do NOT get into relative
merits of the languages that sit on top of these platforms) I've
had way better luck with Perl.

All of the above, of course, is just a personal anecdote designed
to explain my distrust. Take it for what is worth.

> And how do you get a standard java version on each linux distro?

Either java.oracle.com or OpenJDK 7. Both are good choices. In fact
there's quite a bit of progress made by Linux vendors to make
OpenJDK 7 packaging (not just JDK itself, but also the jars/etc)
quite seamless.

Thanks,
Roman.

Re: Python Fork

Posted by Konstantin Boudnik <co...@apache.org>.
I guess the only thing I need to mention about Groovy advantages in this
particular situation is its transparency when it gets to the integration with
_any_ pieces of the Hadoop stack. Case in point is BIGTOP-852.

And the reason for such an ease is Groovy being essentially an improvement of
Java - hence full polyglotism of the too. Wheres with Jython or JRuby you
are talking about a whole different language.

I think I made my point ;)
  Cos

On Thu, Mar 28, 2013 at 09:01AM, Sean Mackrory wrote:
> >> With using puppet and i intend to start a fork of vmbuilder to unify all
> >> these projects in making images of virtual machines. Since vmbuilder is
> >> basically ubuntu only now with centos support floating about in github.
> >> Boxgrinder is kind of gross to get setup because of ruby.
> 
> FYI, I'm most of the way through a proof-of-concept for a tool
> (coincidentally - in Python) that supports the building of a certain VM in
> multiple backends, with Boxgrinder and Kickstarter support right now, and
> hopefully VMBuilder support without much difficulty. As I don't know much
> about VMBuilder yet, would you perhaps want to collaborate on this? I'll be
> posting a patch to BIGTOP-871 in a few days (the one that's there is
> Boxgrinder-specific - I'm working on a much more modular approach).
> 
> Regarding the choice of language, I think I agree with both arguments so
> far - might I suggest Jython? </onlyHalfJoking>. Groovy is already heavily
> used within Bigtop, but it also requires more than just the JVM. I
> personally prefer Python for these purposes - but I wouldn't strongly
> object to anything - as long as it was a well-accepted standard so we don't
> end up with a different language for every little project somebody
> undertakes.
> 
> 
> 
> On Thu, Mar 28, 2013 at 8:04 AM, Philip Herron
> <ph...@wandisco.com>wrote:
> 
> > On 27/03/13 15:37, Roman Shaposhnik wrote:
> > > Hi Philip,
> > >
> > > first of all thanks a bunch for considering to lead with the code --
> > that's
> > > the best way to make progress. I have just started to look at your code
> > > and haven finished reviewing it yet (in fact, I'd rather have somebody
> > > on this list who appreciates Python doing that -- Steve?) but let me
> > > quickly answer some of the points you've raised.
> > >
> > > My only metapoint here would be that since we're stuck with Java
> > > because 90% of Bigtop components are implemented in it
> > > and we've also chosen it as a platform for iTest -- any solution
> > > that leverages Java tooling (Gradle, Groovy, etc) would get
> > > more of my personal enthusiastic embrace than a solution
> > > that introduces yet another platform to care about (Python,
> > > Ruby, etc.).
> >
> > I understand your point here but i think python suits this project much
> > better than a jvm based language because of the dependency of the jvm to
> > me isn't scripting. Python gets by this because its much simpler and
> > installed even in minimal environments of linux by default. And suits
> > this work very well.
> >
> > At the moment we care about:
> >
> > Make/Bash/java/puppet/ruby/kickstart/ From all i can see at the moment,
> > With python we can unify all of bigtop's functionality properly.
> >
> > With using puppet and i intend to start a fork of vmbuilder to unify all
> > these projects in making images of virtual machines. Since vmbuilder is
> > basically ubuntu only now with centos support floating about in github.
> > Boxgrinder is kind of gross to get setup because of ruby.
> >
> > >
> > > On Wed, Mar 27, 2013 at 6:54 AM, Philip Herron
> > > <ph...@wandisco.com> wrote:
> > >> 1 - Bigtop's logic is handled within gnu/make and bash which is
> > >> detrimental for the long term maintainability of bigtop as it may turn
> > >> away developers.
> > >>
> > >> 2 - Make is hard to debug in this way esp for the need of \ for line
> > run-on.
> > >
> > > Could you, please, elaborate on what exact problem you were faced
> > > with when you decided that Python is a better option than Make?
> > > We all have preferences (for example mine is, honestly, make over
> > > python) but of course if the current tooling is inadequate in some
> > > respect it needs to be updated.
> >
> > Yeah I understand you like make i don't mind it but it doesn't have any
> > real long term potential. Because i tend to think most people from java
> > backgrounds will have next to no background with make. I come from a gcc
> > developer/commiter background so i am used to it. But even then this
> > isn't a valid use of make in my opinion and i am sure gnu/make
> > developers would agree this isn't how make should be used.
> >
> > Take for example, the next latest and greatest component wants to be a
> > *.{zip,tar.xz.tar.bz2,.xz,*.7z} this is a basic example but its still
> > valid having the dynamic support for different archives how would we do
> > this in python we could figure this out. In Bash were going to have to
> > have more hooks and more obscure bash to do something like that.
> >
> > >
> > >> 3 - This new directory structure feels more maintainable already with
> > >> bigtop's main function to generate packages for hadoop stack's on
> > >> different platforms. Other functions which (don't seem) to be actively
> > >> maintained like bigtop-deploy and the test framework with regards to
> > >> their documentation.
> > >
> > > Here's my very strong take on it -- without deployment and test Bigtop
> > > might as well pack its stuff and go home. If we're reduced to a
> > "framework
> > > that builds packages" there's very little left in the project to keep me
> > > interested in it.
> >
> > Deployment and test can still be done and there are 100% important i
> > think if i come back with a more complete bigtop with these components
> > my fork might be taken more seriously.
> >
> > >
> > > We *all* should be investing pretty heavily in both. In fact, I hoped
> > > to make Bigtop 0.6.0 about polishing our tests and deployment
> > > code, but Hadoop 2.0.3 story changed that. Hopefully Bigtop 0.7.0
> > > will be a release where we really do invest a lot of energy in
> > > making both squeaky clean.
> > >
> >
> > I understand this, it will be nice to see them.
> >
> > >> 5 - The bigtop.mk BOM file was difficult to read and find errors with
> > >> eval and ?= as well as having archives set in Makefile and package.mk
> > >> doing a lot of tricky bash to get things going.
> > >
> > > Could you, please elaborate? What exactly made it difficult? It is,
> > > essentially, a bunch of key-value pairs (with $substitutions).
> >
> > Yes its key value you pairs but its horrible to read. Coupled with a
> > $eval at the end of each component. I dunno this is a glorified config
> > file and i tend to be very pragmatic about this and think this isn't
> > what it should look like.
> >
> > >
> > >> One thing to note is this isn't any extra dependency as this is just
> > >> stock python code no need for pip install or easy_install of modules.
> > >> And almost all distro's of linux/unix come with python so i think its
> > fine.
> > >
> > > That's my biggest worry -- I had *terrible* luck with Python breaking
> > > between different Linux distributions.
> > >
> > > Thanks,
> > > Roman.
> > >
> >
> > I understand your worry there, this code is pure python no extra depends
> > for now. So that's good and if i get rid of my with usage i can make it
> > python >= 2.4 and this is eliminate this worry dead.
> >
> > Java is by far worse for this problem. example:
> > gcj/eclipse/openjdk/oracle...{5,6,7,8}?
> >
> > And how do you get a standard java version on each linux distro?
> >
> > I will come back with a more solid implementation with deployment and
> > testing.
> >
> > --Phil
> >

Re: Python Fork

Posted by Roman Shaposhnik <rv...@apache.org>.
On Thu, Mar 28, 2013 at 9:01 AM, Sean Mackrory <ma...@gmail.com> wrote:
> I wouldn't strongly object to anything - as long as it was a well-accepted standard
> so we don't end up with a different language for every little project somebody undertakes.

A strong +1 on that!

Thanks,
Roman.

Re: Python Fork

Posted by Sean Mackrory <ma...@gmail.com>.
>> With using puppet and i intend to start a fork of vmbuilder to unify all
>> these projects in making images of virtual machines. Since vmbuilder is
>> basically ubuntu only now with centos support floating about in github.
>> Boxgrinder is kind of gross to get setup because of ruby.

FYI, I'm most of the way through a proof-of-concept for a tool
(coincidentally - in Python) that supports the building of a certain VM in
multiple backends, with Boxgrinder and Kickstarter support right now, and
hopefully VMBuilder support without much difficulty. As I don't know much
about VMBuilder yet, would you perhaps want to collaborate on this? I'll be
posting a patch to BIGTOP-871 in a few days (the one that's there is
Boxgrinder-specific - I'm working on a much more modular approach).

Regarding the choice of language, I think I agree with both arguments so
far - might I suggest Jython? </onlyHalfJoking>. Groovy is already heavily
used within Bigtop, but it also requires more than just the JVM. I
personally prefer Python for these purposes - but I wouldn't strongly
object to anything - as long as it was a well-accepted standard so we don't
end up with a different language for every little project somebody
undertakes.



On Thu, Mar 28, 2013 at 8:04 AM, Philip Herron
<ph...@wandisco.com>wrote:

> On 27/03/13 15:37, Roman Shaposhnik wrote:
> > Hi Philip,
> >
> > first of all thanks a bunch for considering to lead with the code --
> that's
> > the best way to make progress. I have just started to look at your code
> > and haven finished reviewing it yet (in fact, I'd rather have somebody
> > on this list who appreciates Python doing that -- Steve?) but let me
> > quickly answer some of the points you've raised.
> >
> > My only metapoint here would be that since we're stuck with Java
> > because 90% of Bigtop components are implemented in it
> > and we've also chosen it as a platform for iTest -- any solution
> > that leverages Java tooling (Gradle, Groovy, etc) would get
> > more of my personal enthusiastic embrace than a solution
> > that introduces yet another platform to care about (Python,
> > Ruby, etc.).
>
> I understand your point here but i think python suits this project much
> better than a jvm based language because of the dependency of the jvm to
> me isn't scripting. Python gets by this because its much simpler and
> installed even in minimal environments of linux by default. And suits
> this work very well.
>
> At the moment we care about:
>
> Make/Bash/java/puppet/ruby/kickstart/ From all i can see at the moment,
> With python we can unify all of bigtop's functionality properly.
>
> With using puppet and i intend to start a fork of vmbuilder to unify all
> these projects in making images of virtual machines. Since vmbuilder is
> basically ubuntu only now with centos support floating about in github.
> Boxgrinder is kind of gross to get setup because of ruby.
>
> >
> > On Wed, Mar 27, 2013 at 6:54 AM, Philip Herron
> > <ph...@wandisco.com> wrote:
> >> 1 - Bigtop's logic is handled within gnu/make and bash which is
> >> detrimental for the long term maintainability of bigtop as it may turn
> >> away developers.
> >>
> >> 2 - Make is hard to debug in this way esp for the need of \ for line
> run-on.
> >
> > Could you, please, elaborate on what exact problem you were faced
> > with when you decided that Python is a better option than Make?
> > We all have preferences (for example mine is, honestly, make over
> > python) but of course if the current tooling is inadequate in some
> > respect it needs to be updated.
>
> Yeah I understand you like make i don't mind it but it doesn't have any
> real long term potential. Because i tend to think most people from java
> backgrounds will have next to no background with make. I come from a gcc
> developer/commiter background so i am used to it. But even then this
> isn't a valid use of make in my opinion and i am sure gnu/make
> developers would agree this isn't how make should be used.
>
> Take for example, the next latest and greatest component wants to be a
> *.{zip,tar.xz.tar.bz2,.xz,*.7z} this is a basic example but its still
> valid having the dynamic support for different archives how would we do
> this in python we could figure this out. In Bash were going to have to
> have more hooks and more obscure bash to do something like that.
>
> >
> >> 3 - This new directory structure feels more maintainable already with
> >> bigtop's main function to generate packages for hadoop stack's on
> >> different platforms. Other functions which (don't seem) to be actively
> >> maintained like bigtop-deploy and the test framework with regards to
> >> their documentation.
> >
> > Here's my very strong take on it -- without deployment and test Bigtop
> > might as well pack its stuff and go home. If we're reduced to a
> "framework
> > that builds packages" there's very little left in the project to keep me
> > interested in it.
>
> Deployment and test can still be done and there are 100% important i
> think if i come back with a more complete bigtop with these components
> my fork might be taken more seriously.
>
> >
> > We *all* should be investing pretty heavily in both. In fact, I hoped
> > to make Bigtop 0.6.0 about polishing our tests and deployment
> > code, but Hadoop 2.0.3 story changed that. Hopefully Bigtop 0.7.0
> > will be a release where we really do invest a lot of energy in
> > making both squeaky clean.
> >
>
> I understand this, it will be nice to see them.
>
> >> 5 - The bigtop.mk BOM file was difficult to read and find errors with
> >> eval and ?= as well as having archives set in Makefile and package.mk
> >> doing a lot of tricky bash to get things going.
> >
> > Could you, please elaborate? What exactly made it difficult? It is,
> > essentially, a bunch of key-value pairs (with $substitutions).
>
> Yes its key value you pairs but its horrible to read. Coupled with a
> $eval at the end of each component. I dunno this is a glorified config
> file and i tend to be very pragmatic about this and think this isn't
> what it should look like.
>
> >
> >> One thing to note is this isn't any extra dependency as this is just
> >> stock python code no need for pip install or easy_install of modules.
> >> And almost all distro's of linux/unix come with python so i think its
> fine.
> >
> > That's my biggest worry -- I had *terrible* luck with Python breaking
> > between different Linux distributions.
> >
> > Thanks,
> > Roman.
> >
>
> I understand your worry there, this code is pure python no extra depends
> for now. So that's good and if i get rid of my with usage i can make it
> python >= 2.4 and this is eliminate this worry dead.
>
> Java is by far worse for this problem. example:
> gcj/eclipse/openjdk/oracle...{5,6,7,8}?
>
> And how do you get a standard java version on each linux distro?
>
> I will come back with a more solid implementation with deployment and
> testing.
>
> --Phil
>

Re: Python Fork

Posted by Philip Herron <ph...@wandisco.com>.
On 27/03/13 15:37, Roman Shaposhnik wrote:
> Hi Philip,
> 
> first of all thanks a bunch for considering to lead with the code -- that's
> the best way to make progress. I have just started to look at your code
> and haven finished reviewing it yet (in fact, I'd rather have somebody
> on this list who appreciates Python doing that -- Steve?) but let me
> quickly answer some of the points you've raised.
> 
> My only metapoint here would be that since we're stuck with Java
> because 90% of Bigtop components are implemented in it
> and we've also chosen it as a platform for iTest -- any solution
> that leverages Java tooling (Gradle, Groovy, etc) would get
> more of my personal enthusiastic embrace than a solution
> that introduces yet another platform to care about (Python,
> Ruby, etc.).

I understand your point here but i think python suits this project much
better than a jvm based language because of the dependency of the jvm to
me isn't scripting. Python gets by this because its much simpler and
installed even in minimal environments of linux by default. And suits
this work very well.

At the moment we care about:

Make/Bash/java/puppet/ruby/kickstart/ From all i can see at the moment,
With python we can unify all of bigtop's functionality properly.

With using puppet and i intend to start a fork of vmbuilder to unify all
these projects in making images of virtual machines. Since vmbuilder is
basically ubuntu only now with centos support floating about in github.
Boxgrinder is kind of gross to get setup because of ruby.

> 
> On Wed, Mar 27, 2013 at 6:54 AM, Philip Herron
> <ph...@wandisco.com> wrote:
>> 1 - Bigtop's logic is handled within gnu/make and bash which is
>> detrimental for the long term maintainability of bigtop as it may turn
>> away developers.
>>
>> 2 - Make is hard to debug in this way esp for the need of \ for line run-on.
> 
> Could you, please, elaborate on what exact problem you were faced
> with when you decided that Python is a better option than Make?
> We all have preferences (for example mine is, honestly, make over
> python) but of course if the current tooling is inadequate in some
> respect it needs to be updated.

Yeah I understand you like make i don't mind it but it doesn't have any
real long term potential. Because i tend to think most people from java
backgrounds will have next to no background with make. I come from a gcc
developer/commiter background so i am used to it. But even then this
isn't a valid use of make in my opinion and i am sure gnu/make
developers would agree this isn't how make should be used.

Take for example, the next latest and greatest component wants to be a
*.{zip,tar.xz.tar.bz2,.xz,*.7z} this is a basic example but its still
valid having the dynamic support for different archives how would we do
this in python we could figure this out. In Bash were going to have to
have more hooks and more obscure bash to do something like that.

> 
>> 3 - This new directory structure feels more maintainable already with
>> bigtop's main function to generate packages for hadoop stack's on
>> different platforms. Other functions which (don't seem) to be actively
>> maintained like bigtop-deploy and the test framework with regards to
>> their documentation.
> 
> Here's my very strong take on it -- without deployment and test Bigtop
> might as well pack its stuff and go home. If we're reduced to a "framework
> that builds packages" there's very little left in the project to keep me
> interested in it.

Deployment and test can still be done and there are 100% important i
think if i come back with a more complete bigtop with these components
my fork might be taken more seriously.

> 
> We *all* should be investing pretty heavily in both. In fact, I hoped
> to make Bigtop 0.6.0 about polishing our tests and deployment
> code, but Hadoop 2.0.3 story changed that. Hopefully Bigtop 0.7.0
> will be a release where we really do invest a lot of energy in
> making both squeaky clean.
> 

I understand this, it will be nice to see them.

>> 5 - The bigtop.mk BOM file was difficult to read and find errors with
>> eval and ?= as well as having archives set in Makefile and package.mk
>> doing a lot of tricky bash to get things going.
> 
> Could you, please elaborate? What exactly made it difficult? It is,
> essentially, a bunch of key-value pairs (with $substitutions).

Yes its key value you pairs but its horrible to read. Coupled with a
$eval at the end of each component. I dunno this is a glorified config
file and i tend to be very pragmatic about this and think this isn't
what it should look like.

> 
>> One thing to note is this isn't any extra dependency as this is just
>> stock python code no need for pip install or easy_install of modules.
>> And almost all distro's of linux/unix come with python so i think its fine.
> 
> That's my biggest worry -- I had *terrible* luck with Python breaking
> between different Linux distributions.
> 
> Thanks,
> Roman.
> 

I understand your worry there, this code is pure python no extra depends
for now. So that's good and if i get rid of my with usage i can make it
python >= 2.4 and this is eliminate this worry dead.

Java is by far worse for this problem. example:
gcj/eclipse/openjdk/oracle...{5,6,7,8}?

And how do you get a standard java version on each linux distro?

I will come back with a more solid implementation with deployment and
testing.

--Phil

Re: Python Fork

Posted by Roman Shaposhnik <rv...@apache.org>.
Hi Philip,

first of all thanks a bunch for considering to lead with the code -- that's
the best way to make progress. I have just started to look at your code
and haven finished reviewing it yet (in fact, I'd rather have somebody
on this list who appreciates Python doing that -- Steve?) but let me
quickly answer some of the points you've raised.

My only metapoint here would be that since we're stuck with Java
because 90% of Bigtop components are implemented in it
and we've also chosen it as a platform for iTest -- any solution
that leverages Java tooling (Gradle, Groovy, etc) would get
more of my personal enthusiastic embrace than a solution
that introduces yet another platform to care about (Python,
Ruby, etc.).

On Wed, Mar 27, 2013 at 6:54 AM, Philip Herron
<ph...@wandisco.com> wrote:
> 1 - Bigtop's logic is handled within gnu/make and bash which is
> detrimental for the long term maintainability of bigtop as it may turn
> away developers.
>
> 2 - Make is hard to debug in this way esp for the need of \ for line run-on.

Could you, please, elaborate on what exact problem you were faced
with when you decided that Python is a better option than Make?
We all have preferences (for example mine is, honestly, make over
python) but of course if the current tooling is inadequate in some
respect it needs to be updated.

> 3 - This new directory structure feels more maintainable already with
> bigtop's main function to generate packages for hadoop stack's on
> different platforms. Other functions which (don't seem) to be actively
> maintained like bigtop-deploy and the test framework with regards to
> their documentation.

Here's my very strong take on it -- without deployment and test Bigtop
might as well pack its stuff and go home. If we're reduced to a "framework
that builds packages" there's very little left in the project to keep me
interested in it.

We *all* should be investing pretty heavily in both. In fact, I hoped
to make Bigtop 0.6.0 about polishing our tests and deployment
code, but Hadoop 2.0.3 story changed that. Hopefully Bigtop 0.7.0
will be a release where we really do invest a lot of energy in
making both squeaky clean.

> 5 - The bigtop.mk BOM file was difficult to read and find errors with
> eval and ?= as well as having archives set in Makefile and package.mk
> doing a lot of tricky bash to get things going.

Could you, please elaborate? What exactly made it difficult? It is,
essentially, a bunch of key-value pairs (with $substitutions).

> One thing to note is this isn't any extra dependency as this is just
> stock python code no need for pip install or easy_install of modules.
> And almost all distro's of linux/unix come with python so i think its fine.

That's my biggest worry -- I had *terrible* luck with Python breaking
between different Linux distributions.

Thanks,
Roman.

Re: Python Fork

Posted by Bruno Mahé <bm...@apache.org>.
On 03/29/2013 01:42 AM, Bruno Mahé wrote:
> On 03/27/2013 06:54 AM, Philip Herron wrote:
>> Hey all
>>
>> I am new to bigtop and I would like to get feedback on this initial
>> concept from the community on this.
>>
>> I have created a fork of bigtop in my own time at:
>>
>> http://git.buildy.org/?p=bigtop.git;a=shortlog;h=refs/heads/python-fork
>>
>>    $ git clone git://buildy.org/bigtop.git
>>    $ git checkout --track -b python-fork origin/python-fork
>>
>> This fork of bigtop's toplevel Make/Bash logic within python has a
>> number of benefits to the whole community of bigtop for its most
>> important function namely packaging.
>>
>>
>> --Phil
>
> Hi Philip,
>
> I like the direction this is going. But why not using scons? It's in
> python, it's available everywhere and already has support for parallel
> builds.
>
>
> Thanks,
> Bruno

Sorry, I hit the send button a little bit too early.



 >> 3 - This new directory structure feels more maintainable already with
 >> bigtop's main function to generate packages for hadoop stack's on
 >> different platforms. Other functions which (don't seem) to be actively
 >> maintained like bigtop-deploy and the test framework with regards to
 >> their documentation.

bigtop-deploy and the test framework should not be deprecated. They 
receive less attention, but that's no reason to



Codewise:
* I see you import the logging module, but you don't use it. You even 
have your own methods for error/fatal/info.
* nitpick: getopt works, but python has some much nicer argument parsers 
modules :)
* Instead of having a bunch of if RPM elif DEB, I would move these 
implementation details to their own class.
* exit code of subprocess calls should be check. Their output should 
also be displayed and even logged (maybe)


Thanks,
Bruno

Re: Python Fork

Posted by Bruno Mahé <bm...@apache.org>.
On 03/27/2013 06:54 AM, Philip Herron wrote:
> Hey all
>
> I am new to bigtop and I would like to get feedback on this initial
> concept from the community on this.
>
> I have created a fork of bigtop in my own time at:
>
> http://git.buildy.org/?p=bigtop.git;a=shortlog;h=refs/heads/python-fork
>
>    $ git clone git://buildy.org/bigtop.git
>    $ git checkout --track -b python-fork origin/python-fork
>
> This fork of bigtop's toplevel Make/Bash logic within python has a
> number of benefits to the whole community of bigtop for its most
> important function namely packaging.
>
> 1 - Bigtop's logic is handled within gnu/make and bash which is
> detrimental for the long term maintainability of bigtop as it may turn
> away developers.
>
> 2 - Make is hard to debug in this way esp for the need of \ for line run-on.
>
> 3 - This new directory structure feels more maintainable already with
> bigtop's main function to generate packages for hadoop stack's on
> different platforms. Other functions which (don't seem) to be actively
> maintained like bigtop-deploy and the test framework with regards to
> their documentation.
>
> 4 - Using python we can actually take advantage of the system. Using
> threads to do multiple builds of different components at the same time.
> Yes i know we could probably hack together make -j n but this will
> further make things harder to maintain.
>
> 5 - The bigtop.mk BOM file was difficult to read and find errors with
> eval and ?= as well as having archives set in Makefile and package.mk
> doing a lot of tricky bash to get things going.
>
> Using Python ConfigParser we can have a single BOM file for all the
> dynamic data we care about instead of over several makefiles.
>
> ### bigtop.BOM ###
> [DEFAULT]
> APACHE_MIRROR = http://apache.osuosl.org
> APACHE_ARCHIVE = http://archive.apache.org/dist
> ARCHIVES = %(APACHE_MIRROR)s %(APACHE_ARCHIVE)s
>
> [BIGTOP]
> BOM = hadoop
>
> [hadoop]
> NAME = hadoop
> BASE_VERSION = 2.0.3-alpha
> PKG_VERSION = 2.0.3
> RELEASE = 1
> SRC_TYPE = TARGZ
> SRC = %(NAME)s-%(BASE_VERSION)s-src.tar.gz
> DST = %(NAME)s-%(BASE_VERSION)s.tar.gz
> LOC = %(ARCHIVES)s
> DOWNLOAD_PATH = /hadoop/core/%(NAME)s-%(BASE_VERSION)s
> ### EOF ###
>
> I need to spend time documenting but i think this looks very similar but
> cleaner. We have DEFAULTS section to specify our archives then a BOM
> section listing all the components we want in the stack.
>
> The Hadoop component is the only one i have tested with but i don’t see
> any problems adding others. The LOC field in the hadoop component is
> interesting as we can specify the ARCHIVES and it will look in all for
> the tarball we want if the first was down it will go and look into the
> next one if you have several specified.
>
> ----
>
> Over all i could go on and talk about more issues and things that need
> looked at and why this is a good step for bigtop to look at taking but i
> think feedback from the community is more important now.
>
> One thing to note is this isn't any extra dependency as this is just
> stock python code no need for pip install or easy_install of modules.
> And almost all distro's of linux/unix come with python so i think its fine.
>
> Thanks for reading this! I will aim to complete the BOM against trunk
> bigtop and see if i come against anything.
>
> --Phil
>


Hi Philip,

I like the direction this is going. But why not using scons? It's in 
python, it's available everywhere and already has support for parallel 
builds.


Thanks,
Bruno

Re: Python Fork

Posted by Philip Herron <ph...@wandisco.com>.
On 27/03/13 15:26, Sean Mackrory wrote:
> Overall I think I like this - thanks for doing all the work for this fork!
> It would indeed be nice to have a more sophisticated language for this.
> Some thoughts:
> 
>    - If we do make a move to Python, we ought to make sure it works on
>    Python 2.4 initially, as we're still doing releases that we intend to
>    support on RHEL 5. I haven't tried the fork on RHEL 5 yet, so I'm just
>    mentioning it - don't know if there actually are any small issues.
> 
> 
Yeah i could try and and test against this tomorrow there are a few
things i'd have to look out for since i think i used with in there etc
which is python >= 2.5 but its not nessecary etc.

>    - do-component-build and install_.sh: I think those should remain as
>    bash scripts. Did you have any intention to the contrary?
>
Yeah this is true we should deffo keep this this is a nice way to have
the builds be the same on both deb/rpm.

> 
>    - Some of the do-component-build scripts source 'bigtop.bom' to get
>    component versions in the environment. We need to make sure we still
>    generate that in a shell-friendly form.
>
Ah this is an interesting problem but if we were to output environment
variables for the child-processes of subprocess.call this might work.
Other outputting an bash friendly bom like before. There is definitely a
way to work around this. Though i am not sure what way yet :P.

> 
>    - Don't know how I feel about "demoting" the tests. I've been under the
>    impression that they were still maintained (though I admit I haven't paid
>    much attention to them myself), but if they're not well maintained, I think
>    I'd rather we make a plan to give them more attention than accept their
>    unmaintained-ness.
Yeah it might be my ignorance but I’ve never got that side of bigtop to
work yet. And not sure how it works I think I put bigtop-deploy and test
under contrib mostly because it undocumented and not really that
important for the main functions of bigtop to package software.

> 
> Definitely a big decision to make and there's a lot of discussion and work
> to go - but put me down for a +1.
>

And i plan to make the code a lot nicer i did this in an afternoon and i
know its probably not great yet but there is a lot of room for more
interesting use of bigtop for long-term.

Thanks for your +1. I think with some work i could try and bring it up
to scratch with trunk bigtop and see how the community finds it i think
its a necessary step in the long run even though the gcc hacker in my
likes makefiles :).

--Phil


Re: Python Fork

Posted by Sean Mackrory <ma...@gmail.com>.
Overall I think I like this - thanks for doing all the work for this fork!
It would indeed be nice to have a more sophisticated language for this.
Some thoughts:

   - If we do make a move to Python, we ought to make sure it works on
   Python 2.4 initially, as we're still doing releases that we intend to
   support on RHEL 5. I haven't tried the fork on RHEL 5 yet, so I'm just
   mentioning it - don't know if there actually are any small issues.


   - do-component-build and install_.sh: I think those should remain as
   bash scripts. Did you have any intention to the contrary?


   - Some of the do-component-build scripts source 'bigtop.bom' to get
   component versions in the environment. We need to make sure we still
   generate that in a shell-friendly form.


   - Don't know how I feel about "demoting" the tests. I've been under the
   impression that they were still maintained (though I admit I haven't paid
   much attention to them myself), but if they're not well maintained, I think
   I'd rather we make a plan to give them more attention than accept their
   unmaintained-ness.

Definitely a big decision to make and there's a lot of discussion and work
to go - but put me down for a +1.



On Wed, Mar 27, 2013 at 6:54 AM, Philip Herron <ph...@wandisco.com>
wrote:
>
> Hey all
>
> I am new to bigtop and I would like to get feedback on this initial
> concept from the community on this.
>
> I have created a fork of bigtop in my own time at:
>
> http://git.buildy.org/?p=bigtop.git;a=shortlog;h=refs/heads/python-fork
>
>   $ git clone git://buildy.org/bigtop.git
>   $ git checkout --track -b python-fork origin/python-fork
>
> This fork of bigtop's toplevel Make/Bash logic within python has a
> number of benefits to the whole community of bigtop for its most
> important function namely packaging.
>
> 1 - Bigtop's logic is handled within gnu/make and bash which is
> detrimental for the long term maintainability of bigtop as it may turn
> away developers.
>
> 2 - Make is hard to debug in this way esp for the need of \ for line
run-on.
>
> 3 - This new directory structure feels more maintainable already with
> bigtop's main function to generate packages for hadoop stack's on
> different platforms. Other functions which (don't seem) to be actively
> maintained like bigtop-deploy and the test framework with regards to
> their documentation.
>
> 4 - Using python we can actually take advantage of the system. Using
> threads to do multiple builds of different components at the same time.
> Yes i know we could probably hack together make -j n but this will
> further make things harder to maintain.
>
> 5 - The bigtop.mk BOM file was difficult to read and find errors with
> eval and ?= as well as having archives set in Makefile and package.mk
> doing a lot of tricky bash to get things going.
>
> Using Python ConfigParser we can have a single BOM file for all the
> dynamic data we care about instead of over several makefiles.
>
> ### bigtop.BOM ###
> [DEFAULT]
> APACHE_MIRROR = http://apache.osuosl.org
> APACHE_ARCHIVE = http://archive.apache.org/dist
> ARCHIVES = %(APACHE_MIRROR)s %(APACHE_ARCHIVE)s
>
> [BIGTOP]
> BOM = hadoop
>
> [hadoop]
> NAME = hadoop
> BASE_VERSION = 2.0.3-alpha
> PKG_VERSION = 2.0.3
> RELEASE = 1
> SRC_TYPE = TARGZ
> SRC = %(NAME)s-%(BASE_VERSION)s-src.tar.gz
> DST = %(NAME)s-%(BASE_VERSION)s.tar.gz
> LOC = %(ARCHIVES)s
> DOWNLOAD_PATH = /hadoop/core/%(NAME)s-%(BASE_VERSION)s
> ### EOF ###
>
> I need to spend time documenting but i think this looks very similar but
> cleaner. We have DEFAULTS section to specify our archives then a BOM
> section listing all the components we want in the stack.
>
> The Hadoop component is the only one i have tested with but i don’t see
> any problems adding others. The LOC field in the hadoop component is
> interesting as we can specify the ARCHIVES and it will look in all for
> the tarball we want if the first was down it will go and look into the
> next one if you have several specified.
>
> ----
>
> Over all i could go on and talk about more issues and things that need
> looked at and why this is a good step for bigtop to look at taking but i
> think feedback from the community is more important now.
>
> One thing to note is this isn't any extra dependency as this is just
> stock python code no need for pip install or easy_install of modules.
> And almost all distro's of linux/unix come with python so i think its
fine.
>
> Thanks for reading this! I will aim to complete the BOM against trunk
> bigtop and see if i come against anything.
>
> --Phil