[jira] [Comment Edited] (MESOS-898) Transform and audit mesos build process

["Timothy St. Clair (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MESOS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868262#comment-13868262 ] 

Timothy St. Clair edited comment on MESOS-898 at 1/10/14 9:09 PM:
------------------------------------------------------------------

For the purposes of full disclosure, I typically prefer cmake, but that is not the driver in this case.
 
The primary driver is the integration to enable port-esk dependent builds around externals which are "not-found".  There are other side benefits , but they fall into the "nice to have column". (automake 1.14 issues go poof)

e.g. - https://github.com/htcondor/htcondor/blob/master/externals/bundles/coredumper/2011.05.24-r31/CMakeLists.txt  

Translation won't be that difficult for me, but there is indeed a mind-shift to deal with the differences. 

I think if we take an agnostic approach and lay out requirements, it will set the expectations, then tool choice will be driven by requirements.  From my perspective I see the requirements as follows: 

Primary:
1.) Remove tars and .zips from source tree to make patching manageable. 
2.) Enable both a proper and non-proper build (old style build against targeted deps, or system deps)
3.) Subsume libprocess and stout into mesos itself (I've done this in my patches).
4.) Ensure all targets are properly installed and in standard system locations.
5.) Harden code base to handle multiple versions of libraries.

Secondary:
1.) Ensure make -jX always works (didn't work for me)
2.) Ensure out of source builds work
3.) Ensure ccache works properly 
4.) Ensure dep-graph is complete and proper.  (I think this why -jX fails and it rebuilds a lot on back-to-back 'make check' runs) 
...


was (Author: tstclair):
For the purposes of full disclosure, I typically prefer cmake, but that is not the driver in this case.
 
The primary driver is the integration to enable port-esk dependent builds around externals which are "not-found".  There are other side benefits , but they fall into the "nice to have column". (automake 1.14 issues go poof)

e.g. - https://github.com/htcondor/htcondor/blob/master/externals/bundles/coredumper/2011.05.24-r31/CMakeLists.txt  

Translation won't be that difficult for me, but there is indeed a mind-shift to deal with the differences. 

I think if we take an agnostic approach and lay out requirements, it will set the expectations, then tool choice will be driven by requirements.  From my perspective I see the requirements as follows: 

Primary:
1.) Remove tars and .zips from source tree to make patching manageable. 
2.) Enable both a proper and non-proper build (old style build against targeted deps, or system deps)
2.) Subsume libprocess and stout into mesos itself (I've done this in my patches).
3.) Ensure all targets are properly installed and in standard system locations.
4.) Harden code base to handle multiple versions of libraries.

Secondary:
1.) Ensure make -jX always works (didn't work for me)
2.) Ensure out of source builds work
3.) Ensure ccache works properly 
4.) Ensure dep-graph is complete and proper.  (I think this why -jX fails and it rebuilds a lot on back-to-back 'make check' runs) 
...

> Transform and audit mesos build process
> ---------------------------------------
>
>                 Key: MESOS-898
>                 URL: https://issues.apache.org/jira/browse/MESOS-898
>             Project: Mesos
>          Issue Type: Improvement
>          Components: build
>            Reporter: Timothy St. Clair
>              Labels: build
>
> This is a rather substantial undertaking, so I would want upstream debate+buy-in prior to full commitment.  The basic premise is: upstream rebundles several of its dependencies in part to tightly control its stack.  This is not out of the norm, but in order to be picked up by distribution channels it needs to built against system dependencies, and rebundling is strictly forbidden.  Given that the mesos primary target platform are data-center distributions such as RHEL/CENTOS/SL it makes sense to still have bundling support for those who do not have dependencies in their channels "yet".  This is where cmake can be win with it's uber macros (http://www.cmake.org/cmake/help/v2.8.8/cmake.html#module:ExternalProject).  I do not know of any equivalent in the autotools world, other then to brew your own solution.   I've done this type of work in the past, and completely transformed condor and would leverage a lot of the work that was done there. 
> I currently have a tracking branch where I've started this work, but before I go off into the woods, it makes sense to have a debate in public. 
> The primary benefits are: 
> 1. Enable downstream channels to easily distro without carrying a large patch sets. 
> 2. Still support existing "non-proper" distribution methods. 
> 3. Harden / future proof dependent interfaces. 
> Side Benefits: 
> Audit current build mechanics.  
>  - Presently the language specific binding are not installed.  (.py & .jar)
>  - make -jX currently fails 
>  - optionally look in arm support. 
> Costs:
> 1. Time
> 2. Potential temporary destabilization
> 3. Infrastructure around build+test may need to change.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)