You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by an...@schwartzmeyer.com on 2017/08/25 18:47:53 UTC

CMake build refactoring

Hello Mesos developers,

I wanted to take the time to announce a (long) patch chain that fixes up 
the CMake build system. There were a lot of open bugs due to the initial 
design (and constraints of CMake 2), and by moving to CMake 3, I was 
able to rewrite the CMake build in a target-based dependency graph 
manner (the way CMake is meant to be used).

The very end of the chain is here: https://reviews.apache.org/r/61753/

Joseph Wu is currently shepherding it. That final patch includes the 
what and the why explanation of this patch series.

My main guideline in this effort was MESOS-3576, which pointed out the 
incorrectness of the link flags generated by CMake. This has been 
resolved, and moreover, a significant amount of superfluous CMake code 
was deleted (that final patch was about a 4 to 1 ratio of deletions to 
adds).

I took a two-phase approach to this reworking. The first phase imported 
all of our third party dependencies correctly. Instead of linking to 
these dependencies by manually including their various include/library 
directories and compilation/link flags, each dependency was added to the 
CMake graph as an imported target. All necessary information for linking 
to the dependency is recorded in its target, and used is transitively 
passed to targets which take it as dependency by CMake. Adding a new 
dependency is now relatively simple, with many easy-to-replicate 
patterns available in 3rdparty/CMakeLists.txt.

An important note is that this was also done for header-only libraries, 
including stout. CMake 3 has a notion of "interface libraries" where a 
header-only library can be represented as a CMake target like any other 
library. With this done correctly, linking stout to boost is as simple 
as `target_link_libraries(stout boost)`, and linking libprocess to stout 
is `target_link_libraries(process stout)`, and the boost dependency is 
understood transitively.

The second phase was refactoring the Mesos build itself (that is, not 
the third party dependencies). With all our dependencies imported 
properly, I was able to delete the files of extraneous information such 
as `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This 
information is instead correctly stored in the aforementioned CMake 
dependency graph.

With this all put together, the entirety of the required CMake code to 
build the agent executable, on all platforms, is the following _two 
lines_:

     add_executable(mesos-agent main.cpp)
     target_link_libraries(mesos-agent PRIVATE mesos)

All necessary link and compilation flags are parsed by CMake through the 
dependency graph as it visits libmesos as the mesos-agent dependency.

I keep an updated tree on GitHub here for easy testing: 
https://github.com/andschwa/mesos/blob/cmake-refactor/src/slave/CMakeLists.txt

As we move closer to deprecating Autotools, I wanted to ensure that the 
replacement build system was as correct and easy-to-use as possible. As 
with any build system (and any software engineering effort), there is 
always more to clean up and improve. However, I am satisfied with the 
result of my efforts, and I hope this build system makes your work as 
developers easier. If you have the time, please take a look at the the 
patches and give it a test. Let me know know I can make it work better 
for you.

Thank you,

Andrew Schwartzmeyer

RE: CMake build refactoring

Posted by Jeff Coffler <Je...@microsoft.com.INVALID>.
Looks like Andy didn't answer this, so I will.

Currently Linux builds still use autotools but can use cmake. Buildbot uses autotools for Linux. Windows builds exclusively uses cmake.

End result: For now, reviewers should insure that changes which affect files (new source files, new header files for autotools) should work in both autoconf and cmake. This will continue until final deprecation of autotools. I'm unsure when that will be, but it'll be a while.

If you don't do that, then you will break some build (and some buildbot).

/Jeff

-----Original Message-----
From: Benno Evers [mailto:bevers@mesosphere.com] 
Sent: Monday, August 28, 2017 2:41 AM
To: dev@mesos.apache.org
Subject: Re: CMake build refactoring

Hi Andrew,

thanks for working on this, it looks very nice.

Is there a policy on how to handle changes to the build system before the official deprecation of autotools? I.e., should should reviewers require that new features are added only to autoconf, only to CMake, or to both?

Thanks,
Benno

On Sat, Aug 26, 2017 at 2:39 AM, <an...@schwartzmeyer.com> wrote:

> Hi Zhitao,
>
> 1) I talked with Joe and confirmed that we're targeting 1.5 for this chain.
>
> 2) For Mesos modules, the included example modules are built, but I 
> don't have an external example. I think the easiest approach to doing 
> so is to include Mesos as a submodule and then 
> `add_subdirectory(mesos)`, which would bring the target for `libmesos` 
> into your graph to build the module against. I can work with anyone willing to create an example.
>
> 3) Yes, that is the plan. The JIRA issue tracking CMake is MESOS-898, 
> and the last thread about the (eventual) deprecation of Autotools is here:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail-
> archives.apache.org%2Fmod_mbox%2Fmesos-dev%2F201703.mbox%2Fbrowser&dat
> a=02%7C01%7CJeff.Coffler%40microsoft.com%7Ceaf570e831614a3bac2608d4edf
> 8fc46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636395101029885531&
> sdata=vqqfe%2BQJqOoh6ZMp15W4zHO0iNj3WFYM7nkRU6ovgxE%3D&reserved=0
>
> Thanks,
>
> Andrew
>
>
> On 08/25/2017 2:40 pm, Zhitao Li wrote:
>
>> Hi Andrew,
>>
>> Thank you for the great work. Really looking forward to see a more 
>> structured dependency graph.
>>
>> A couple of questions:
>>
>> 1) is there a target version of this work? Will the chain land on 1.5?
>>
>> 2) For people who maintains Mesos modules, is there a example build 
>> of how people can arrange their build?
>>
>> 3) Is there plan to make CMake the only supported build system? If 
>> yes, what's the graduation standard?
>>
>> Thanks!
>>
>> On Fri, Aug 25, 2017 at 1:00 PM, Vinod Kone <vi...@apache.org> wrote:
>>
>> +1
>>>
>>> On Fri, Aug 25, 2017 at 12:08 PM, <an...@schwartzmeyer.com> wrote:
>>>
>>> > I have the Java libraries building now on both Linux and Windows
>>> (though
>>> > the initial setup on Windows is a bit annoying to get all the 
>>> > tools in
>>> the
>>> > right place, I'd like to follow up this chain with a 
>>> > `docs/cmake.md`
>>> with
>>> > all the odd little bits of information).
>>> >
>>> > I also enabled the ZooKeeper unit tests (which I needed in order 
>>> > to integration test the patches that went upstream to ZooKeeper to 
>>> > give
>>> their
>>> > C Client library a CMake build, too).
>>> >
>>> > Furthermore, with the Java library, I've successfully run Marathon
>>> against
>>> > CMake built Mesos libraries on Linux.
>>> >
>>> > I do not have the Python libraries building yet, and there is a 
>>> > "Java examples" library that is not yet built. I'll make sure to 
>>> > file issues
>>> to
>>> > update these.
>>> >
>>> > Joe is working to update the Mesosphere CI, I will check with him 
>>> > about updating `docker-build.sh`.
>>> >
>>> > Thanks Vinod!
>>> >
>>> >
>>> > On 08/25/2017 11:59 am, Vinod Kone wrote:
>>> >
>>> >> This is great to hear. Thanks for your perseverance on making 
>>> >> this
>>> happen
>>> >> Andy!
>>> >>
>>> >> Btw, do we build Java and Python libraries too now?
>>> >>
>>> >> Once we land the CMake changes, lets make sure to update 
>>> >> `support/docker-build.sh` to make sure it is continuously being 
>>> >> tested
>>> on
>>> >> our CI.
>>> >>
>>> >> On Fri, Aug 25, 2017 at 11:47 AM, <an...@schwartzmeyer.com> wrote:
>>> >>
>>> >> Hello Mesos developers,
>>> >>>
>>> >>> I wanted to take the time to announce a (long) patch chain that 
>>> >>> fixes
>>> up
>>> >>> the CMake build system. There were a lot of open bugs due to the
>>> initial
>>> >>> design (and constraints of CMake 2), and by moving to CMake 3, I 
>>> >>> was
>>> able
>>> >>> to rewrite the CMake build in a target-based dependency graph 
>>> >>> manner
>>> (the
>>> >>> way CMake is meant to be used).
>>> >>>
>>> >>> The very end of the chain is here: 
>>> >>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>> >>> Freviews.apache.org%2Fr%2F6&data=02%7C01%7CJeff.Coffler%40micros
>>> >>> oft.com%7Ceaf570e831614a3bac2608d4edf8fc46%7C72f988bf86f141af91a
>>> >>> b2d7cd011db47%7C1%7C0%7C636395101029885531&sdata=V6dxdCiuNSlhWQL
>>> >>> hkE2k1R3coDGVBPWz1p1XymuVxMM%3D&reserved=0
>>> 1753/
>>> >>>
>>> >>> Joseph Wu is currently shepherding it. That final patch includes 
>>> >>> the
>>> what
>>> >>> and the why explanation of this patch series.
>>> >>>
>>> >>> My main guideline in this effort was MESOS-3576, which pointed 
>>> >>> out
>>> the
>>> >>> incorrectness of the link flags generated by CMake. This has 
>>> >>> been resolved, and moreover, a significant amount of superfluous 
>>> >>> CMake code was
>>> deleted
>>> >>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>>> >>>
>>> >>> I took a two-phase approach to this reworking. The first phase
>>> imported
>>> >>> all of our third party dependencies correctly. Instead of 
>>> >>> linking to these dependencies by manually including their 
>>> >>> various include/library directories and compilation/link flags, 
>>> >>> each dependency was added to
>>> the
>>> >>> CMake graph as an imported target. All necessary information for
>>> linking
>>> >>> to
>>> >>> the dependency is recorded in its target, and used is 
>>> >>> transitively
>>> passed
>>> >>> to targets which take it as dependency by CMake. Adding a new
>>> dependency
>>> >>> is
>>> >>> now relatively simple, with many easy-to-replicate patterns 
>>> >>> available
>>> in
>>> >>> 3rdparty/CMakeLists.txt.
>>> >>>
>>> >>> An important note is that this was also done for header-only
>>> libraries,
>>> >>> including stout. CMake 3 has a notion of "interface libraries" 
>>> >>> where
>>> a
>>> >>> header-only library can be represented as a CMake target like 
>>> >>> any
>>> other
>>> >>> library. With this done correctly, linking stout to boost is as
>>> simple
>>> as
>>> >>> `target_link_libraries(stout boost)`, and linking libprocess to 
>>> >>> stout
>>> is
>>> >>> `target_link_libraries(process stout)`, and the boost dependency 
>>> >>> is understood transitively.
>>> >>>
>>> >>> The second phase was refactoring the Mesos build itself (that 
>>> >>> is, not
>>> the
>>> >>> third party dependencies). With all our dependencies imported
>>> properly, I
>>> >>> was able to delete the files of extraneous information such as 
>>> >>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This
>>> information
>>> is
>>> >>> instead correctly stored in the aforementioned CMake dependency
>>> graph.
>>> >>>
>>> >>> With this all put together, the entirety of the required CMake 
>>> >>> code
>>> to
>>> >>> build the agent executable, on all platforms, is the following 
>>> >>> _two
>>> >>> lines_:
>>> >>>
>>> >>>     add_executable(mesos-agent main.cpp)
>>> >>>     target_link_libraries(mesos-agent PRIVATE mesos)
>>> >>>
>>> >>> All necessary link and compilation flags are parsed by CMake 
>>> >>> through
>>> the
>>> >>> dependency graph as it visits libmesos as the mesos-agent dependency.
>>> >>>
>>> >>> I keep an updated tree on GitHub here for easy testing:
>>> >>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>> >>> Fgithub.com%2Fandschwa%2Fmesos%2Fblob%2Fcmake-refactor%2Fsrc%2Fs
>>> >>> l&data=02%7C01%7CJeff.Coffler%40microsoft.com%7Ceaf570e831614a3b
>>> >>> ac2608d4edf8fc46%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63
>>> >>> 6395101029885531&sdata=rNiY2lyXnOTDv2TvKdhHjBIYcJSXqebZTfTsqWasR
>>> >>> sE%3D&reserved=0
>>> >>> ave/CMakeLists.txt
>>> >>>
>>> >>> As we move closer to deprecating Autotools, I wanted to ensure 
>>> >>> that
>>> the
>>> >>> replacement build system was as correct and easy-to-use as possible.
>>> As
>>> >>> with any build system (and any software engineering effort), 
>>> >>> there is always more to clean up and improve. However, I am 
>>> >>> satisfied with the result of my efforts, and I hope this build 
>>> >>> system makes your work as developers easier. If you have the 
>>> >>> time, please take a look at the
>>> the
>>> >>> patches and give it a test. Let me know know I can make it work
>>> better
>>> >>> for
>>> >>> you.
>>> >>>
>>> >>> Thank you,
>>> >>>
>>> >>> Andrew Schwartzmeyer
>>> >>>
>>> >>>
>>>
>>>


--
Benno Evers
Software Engineer, Mesosphere

Re: CMake build refactoring

Posted by Benno Evers <be...@mesosphere.com>.
Hi Andrew,

thanks for working on this, it looks very nice.

Is there a policy on how to handle changes to the build system before the
official deprecation of autotools? I.e., should should reviewers require
that new features are added only to autoconf, only to CMake, or to both?

Thanks,
Benno

On Sat, Aug 26, 2017 at 2:39 AM, <an...@schwartzmeyer.com> wrote:

> Hi Zhitao,
>
> 1) I talked with Joe and confirmed that we're targeting 1.5 for this chain.
>
> 2) For Mesos modules, the included example modules are built, but I don't
> have an external example. I think the easiest approach to doing so is to
> include Mesos as a submodule and then `add_subdirectory(mesos)`, which
> would bring the target for `libmesos` into your graph to build the module
> against. I can work with anyone willing to create an example.
>
> 3) Yes, that is the plan. The JIRA issue tracking CMake is MESOS-898, and
> the last thread about the (eventual) deprecation of Autotools is here:
> https://mail-archives.apache.org/mod_mbox/mesos-dev/201703.mbox/browser
>
> Thanks,
>
> Andrew
>
>
> On 08/25/2017 2:40 pm, Zhitao Li wrote:
>
>> Hi Andrew,
>>
>> Thank you for the great work. Really looking forward to see a more
>> structured dependency graph.
>>
>> A couple of questions:
>>
>> 1) is there a target version of this work? Will the chain land on 1.5?
>>
>> 2) For people who maintains Mesos modules, is there a example build of how
>> people can arrange their build?
>>
>> 3) Is there plan to make CMake the only supported build system? If yes,
>> what's the graduation standard?
>>
>> Thanks!
>>
>> On Fri, Aug 25, 2017 at 1:00 PM, Vinod Kone <vi...@apache.org> wrote:
>>
>> +1
>>>
>>> On Fri, Aug 25, 2017 at 12:08 PM, <an...@schwartzmeyer.com> wrote:
>>>
>>> > I have the Java libraries building now on both Linux and Windows
>>> (though
>>> > the initial setup on Windows is a bit annoying to get all the tools in
>>> the
>>> > right place, I'd like to follow up this chain with a `docs/cmake.md`
>>> with
>>> > all the odd little bits of information).
>>> >
>>> > I also enabled the ZooKeeper unit tests (which I needed in order to
>>> > integration test the patches that went upstream to ZooKeeper to give
>>> their
>>> > C Client library a CMake build, too).
>>> >
>>> > Furthermore, with the Java library, I've successfully run Marathon
>>> against
>>> > CMake built Mesos libraries on Linux.
>>> >
>>> > I do not have the Python libraries building yet, and there is a "Java
>>> > examples" library that is not yet built. I'll make sure to file issues
>>> to
>>> > update these.
>>> >
>>> > Joe is working to update the Mesosphere CI, I will check with him about
>>> > updating `docker-build.sh`.
>>> >
>>> > Thanks Vinod!
>>> >
>>> >
>>> > On 08/25/2017 11:59 am, Vinod Kone wrote:
>>> >
>>> >> This is great to hear. Thanks for your perseverance on making this
>>> happen
>>> >> Andy!
>>> >>
>>> >> Btw, do we build Java and Python libraries too now?
>>> >>
>>> >> Once we land the CMake changes, lets make sure to update
>>> >> `support/docker-build.sh` to make sure it is continuously being tested
>>> on
>>> >> our CI.
>>> >>
>>> >> On Fri, Aug 25, 2017 at 11:47 AM, <an...@schwartzmeyer.com> wrote:
>>> >>
>>> >> Hello Mesos developers,
>>> >>>
>>> >>> I wanted to take the time to announce a (long) patch chain that fixes
>>> up
>>> >>> the CMake build system. There were a lot of open bugs due to the
>>> initial
>>> >>> design (and constraints of CMake 2), and by moving to CMake 3, I was
>>> able
>>> >>> to rewrite the CMake build in a target-based dependency graph manner
>>> (the
>>> >>> way CMake is meant to be used).
>>> >>>
>>> >>> The very end of the chain is here: https://reviews.apache.org/r/6
>>> 1753/
>>> >>>
>>> >>> Joseph Wu is currently shepherding it. That final patch includes the
>>> what
>>> >>> and the why explanation of this patch series.
>>> >>>
>>> >>> My main guideline in this effort was MESOS-3576, which pointed out
>>> the
>>> >>> incorrectness of the link flags generated by CMake. This has been
>>> >>> resolved,
>>> >>> and moreover, a significant amount of superfluous CMake code was
>>> deleted
>>> >>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>>> >>>
>>> >>> I took a two-phase approach to this reworking. The first phase
>>> imported
>>> >>> all of our third party dependencies correctly. Instead of linking to
>>> >>> these
>>> >>> dependencies by manually including their various include/library
>>> >>> directories and compilation/link flags, each dependency was added to
>>> the
>>> >>> CMake graph as an imported target. All necessary information for
>>> linking
>>> >>> to
>>> >>> the dependency is recorded in its target, and used is transitively
>>> passed
>>> >>> to targets which take it as dependency by CMake. Adding a new
>>> dependency
>>> >>> is
>>> >>> now relatively simple, with many easy-to-replicate patterns available
>>> in
>>> >>> 3rdparty/CMakeLists.txt.
>>> >>>
>>> >>> An important note is that this was also done for header-only
>>> libraries,
>>> >>> including stout. CMake 3 has a notion of "interface libraries" where
>>> a
>>> >>> header-only library can be represented as a CMake target like any
>>> other
>>> >>> library. With this done correctly, linking stout to boost is as
>>> simple
>>> as
>>> >>> `target_link_libraries(stout boost)`, and linking libprocess to stout
>>> is
>>> >>> `target_link_libraries(process stout)`, and the boost dependency is
>>> >>> understood transitively.
>>> >>>
>>> >>> The second phase was refactoring the Mesos build itself (that is, not
>>> the
>>> >>> third party dependencies). With all our dependencies imported
>>> properly, I
>>> >>> was able to delete the files of extraneous information such as
>>> >>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This
>>> information
>>> is
>>> >>> instead correctly stored in the aforementioned CMake dependency
>>> graph.
>>> >>>
>>> >>> With this all put together, the entirety of the required CMake code
>>> to
>>> >>> build the agent executable, on all platforms, is the following _two
>>> >>> lines_:
>>> >>>
>>> >>>     add_executable(mesos-agent main.cpp)
>>> >>>     target_link_libraries(mesos-agent PRIVATE mesos)
>>> >>>
>>> >>> All necessary link and compilation flags are parsed by CMake through
>>> the
>>> >>> dependency graph as it visits libmesos as the mesos-agent dependency.
>>> >>>
>>> >>> I keep an updated tree on GitHub here for easy testing:
>>> >>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
>>> >>> ave/CMakeLists.txt
>>> >>>
>>> >>> As we move closer to deprecating Autotools, I wanted to ensure that
>>> the
>>> >>> replacement build system was as correct and easy-to-use as possible.
>>> As
>>> >>> with any build system (and any software engineering effort), there is
>>> >>> always more to clean up and improve. However, I am satisfied with the
>>> >>> result of my efforts, and I hope this build system makes your work as
>>> >>> developers easier. If you have the time, please take a look at the
>>> the
>>> >>> patches and give it a test. Let me know know I can make it work
>>> better
>>> >>> for
>>> >>> you.
>>> >>>
>>> >>> Thank you,
>>> >>>
>>> >>> Andrew Schwartzmeyer
>>> >>>
>>> >>>
>>>
>>>


-- 
Benno Evers
Software Engineer, Mesosphere

Re: CMake build refactoring

Posted by an...@schwartzmeyer.com.
Hi Zhitao,

1) I talked with Joe and confirmed that we're targeting 1.5 for this 
chain.

2) For Mesos modules, the included example modules are built, but I 
don't have an external example. I think the easiest approach to doing so 
is to include Mesos as a submodule and then `add_subdirectory(mesos)`, 
which would bring the target for `libmesos` into your graph to build the 
module against. I can work with anyone willing to create an example.

3) Yes, that is the plan. The JIRA issue tracking CMake is MESOS-898, 
and the last thread about the (eventual) deprecation of Autotools is 
here: 
https://mail-archives.apache.org/mod_mbox/mesos-dev/201703.mbox/browser

Thanks,

Andrew

On 08/25/2017 2:40 pm, Zhitao Li wrote:
> Hi Andrew,
> 
> Thank you for the great work. Really looking forward to see a more
> structured dependency graph.
> 
> A couple of questions:
> 
> 1) is there a target version of this work? Will the chain land on 1.5?
> 
> 2) For people who maintains Mesos modules, is there a example build of 
> how
> people can arrange their build?
> 
> 3) Is there plan to make CMake the only supported build system? If yes,
> what's the graduation standard?
> 
> Thanks!
> 
> On Fri, Aug 25, 2017 at 1:00 PM, Vinod Kone <vi...@apache.org> 
> wrote:
> 
>> +1
>> 
>> On Fri, Aug 25, 2017 at 12:08 PM, <an...@schwartzmeyer.com> wrote:
>> 
>> > I have the Java libraries building now on both Linux and Windows (though
>> > the initial setup on Windows is a bit annoying to get all the tools in
>> the
>> > right place, I'd like to follow up this chain with a `docs/cmake.md`
>> with
>> > all the odd little bits of information).
>> >
>> > I also enabled the ZooKeeper unit tests (which I needed in order to
>> > integration test the patches that went upstream to ZooKeeper to give
>> their
>> > C Client library a CMake build, too).
>> >
>> > Furthermore, with the Java library, I've successfully run Marathon
>> against
>> > CMake built Mesos libraries on Linux.
>> >
>> > I do not have the Python libraries building yet, and there is a "Java
>> > examples" library that is not yet built. I'll make sure to file issues to
>> > update these.
>> >
>> > Joe is working to update the Mesosphere CI, I will check with him about
>> > updating `docker-build.sh`.
>> >
>> > Thanks Vinod!
>> >
>> >
>> > On 08/25/2017 11:59 am, Vinod Kone wrote:
>> >
>> >> This is great to hear. Thanks for your perseverance on making this
>> happen
>> >> Andy!
>> >>
>> >> Btw, do we build Java and Python libraries too now?
>> >>
>> >> Once we land the CMake changes, lets make sure to update
>> >> `support/docker-build.sh` to make sure it is continuously being tested
>> on
>> >> our CI.
>> >>
>> >> On Fri, Aug 25, 2017 at 11:47 AM, <an...@schwartzmeyer.com> wrote:
>> >>
>> >> Hello Mesos developers,
>> >>>
>> >>> I wanted to take the time to announce a (long) patch chain that fixes
>> up
>> >>> the CMake build system. There were a lot of open bugs due to the
>> initial
>> >>> design (and constraints of CMake 2), and by moving to CMake 3, I was
>> able
>> >>> to rewrite the CMake build in a target-based dependency graph manner
>> (the
>> >>> way CMake is meant to be used).
>> >>>
>> >>> The very end of the chain is here: https://reviews.apache.org/r/61753/
>> >>>
>> >>> Joseph Wu is currently shepherding it. That final patch includes the
>> what
>> >>> and the why explanation of this patch series.
>> >>>
>> >>> My main guideline in this effort was MESOS-3576, which pointed out the
>> >>> incorrectness of the link flags generated by CMake. This has been
>> >>> resolved,
>> >>> and moreover, a significant amount of superfluous CMake code was
>> deleted
>> >>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>> >>>
>> >>> I took a two-phase approach to this reworking. The first phase imported
>> >>> all of our third party dependencies correctly. Instead of linking to
>> >>> these
>> >>> dependencies by manually including their various include/library
>> >>> directories and compilation/link flags, each dependency was added to
>> the
>> >>> CMake graph as an imported target. All necessary information for
>> linking
>> >>> to
>> >>> the dependency is recorded in its target, and used is transitively
>> passed
>> >>> to targets which take it as dependency by CMake. Adding a new
>> dependency
>> >>> is
>> >>> now relatively simple, with many easy-to-replicate patterns available
>> in
>> >>> 3rdparty/CMakeLists.txt.
>> >>>
>> >>> An important note is that this was also done for header-only libraries,
>> >>> including stout. CMake 3 has a notion of "interface libraries" where a
>> >>> header-only library can be represented as a CMake target like any other
>> >>> library. With this done correctly, linking stout to boost is as simple
>> as
>> >>> `target_link_libraries(stout boost)`, and linking libprocess to stout
>> is
>> >>> `target_link_libraries(process stout)`, and the boost dependency is
>> >>> understood transitively.
>> >>>
>> >>> The second phase was refactoring the Mesos build itself (that is, not
>> the
>> >>> third party dependencies). With all our dependencies imported
>> properly, I
>> >>> was able to delete the files of extraneous information such as
>> >>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This information
>> is
>> >>> instead correctly stored in the aforementioned CMake dependency graph.
>> >>>
>> >>> With this all put together, the entirety of the required CMake code to
>> >>> build the agent executable, on all platforms, is the following _two
>> >>> lines_:
>> >>>
>> >>>     add_executable(mesos-agent main.cpp)
>> >>>     target_link_libraries(mesos-agent PRIVATE mesos)
>> >>>
>> >>> All necessary link and compilation flags are parsed by CMake through
>> the
>> >>> dependency graph as it visits libmesos as the mesos-agent dependency.
>> >>>
>> >>> I keep an updated tree on GitHub here for easy testing:
>> >>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
>> >>> ave/CMakeLists.txt
>> >>>
>> >>> As we move closer to deprecating Autotools, I wanted to ensure that the
>> >>> replacement build system was as correct and easy-to-use as possible. As
>> >>> with any build system (and any software engineering effort), there is
>> >>> always more to clean up and improve. However, I am satisfied with the
>> >>> result of my efforts, and I hope this build system makes your work as
>> >>> developers easier. If you have the time, please take a look at the the
>> >>> patches and give it a test. Let me know know I can make it work better
>> >>> for
>> >>> you.
>> >>>
>> >>> Thank you,
>> >>>
>> >>> Andrew Schwartzmeyer
>> >>>
>> >>>
>> 

Re: CMake build refactoring

Posted by Zhitao Li <zh...@gmail.com>.
Hi Andrew,

Thank you for the great work. Really looking forward to see a more
structured dependency graph.

A couple of questions:

1) is there a target version of this work? Will the chain land on 1.5?

2) For people who maintains Mesos modules, is there a example build of how
people can arrange their build?

3) Is there plan to make CMake the only supported build system? If yes,
what's the graduation standard?

Thanks!

On Fri, Aug 25, 2017 at 1:00 PM, Vinod Kone <vi...@apache.org> wrote:

> +1
>
> On Fri, Aug 25, 2017 at 12:08 PM, <an...@schwartzmeyer.com> wrote:
>
> > I have the Java libraries building now on both Linux and Windows (though
> > the initial setup on Windows is a bit annoying to get all the tools in
> the
> > right place, I'd like to follow up this chain with a `docs/cmake.md`
> with
> > all the odd little bits of information).
> >
> > I also enabled the ZooKeeper unit tests (which I needed in order to
> > integration test the patches that went upstream to ZooKeeper to give
> their
> > C Client library a CMake build, too).
> >
> > Furthermore, with the Java library, I've successfully run Marathon
> against
> > CMake built Mesos libraries on Linux.
> >
> > I do not have the Python libraries building yet, and there is a "Java
> > examples" library that is not yet built. I'll make sure to file issues to
> > update these.
> >
> > Joe is working to update the Mesosphere CI, I will check with him about
> > updating `docker-build.sh`.
> >
> > Thanks Vinod!
> >
> >
> > On 08/25/2017 11:59 am, Vinod Kone wrote:
> >
> >> This is great to hear. Thanks for your perseverance on making this
> happen
> >> Andy!
> >>
> >> Btw, do we build Java and Python libraries too now?
> >>
> >> Once we land the CMake changes, lets make sure to update
> >> `support/docker-build.sh` to make sure it is continuously being tested
> on
> >> our CI.
> >>
> >> On Fri, Aug 25, 2017 at 11:47 AM, <an...@schwartzmeyer.com> wrote:
> >>
> >> Hello Mesos developers,
> >>>
> >>> I wanted to take the time to announce a (long) patch chain that fixes
> up
> >>> the CMake build system. There were a lot of open bugs due to the
> initial
> >>> design (and constraints of CMake 2), and by moving to CMake 3, I was
> able
> >>> to rewrite the CMake build in a target-based dependency graph manner
> (the
> >>> way CMake is meant to be used).
> >>>
> >>> The very end of the chain is here: https://reviews.apache.org/r/61753/
> >>>
> >>> Joseph Wu is currently shepherding it. That final patch includes the
> what
> >>> and the why explanation of this patch series.
> >>>
> >>> My main guideline in this effort was MESOS-3576, which pointed out the
> >>> incorrectness of the link flags generated by CMake. This has been
> >>> resolved,
> >>> and moreover, a significant amount of superfluous CMake code was
> deleted
> >>> (that final patch was about a 4 to 1 ratio of deletions to adds).
> >>>
> >>> I took a two-phase approach to this reworking. The first phase imported
> >>> all of our third party dependencies correctly. Instead of linking to
> >>> these
> >>> dependencies by manually including their various include/library
> >>> directories and compilation/link flags, each dependency was added to
> the
> >>> CMake graph as an imported target. All necessary information for
> linking
> >>> to
> >>> the dependency is recorded in its target, and used is transitively
> passed
> >>> to targets which take it as dependency by CMake. Adding a new
> dependency
> >>> is
> >>> now relatively simple, with many easy-to-replicate patterns available
> in
> >>> 3rdparty/CMakeLists.txt.
> >>>
> >>> An important note is that this was also done for header-only libraries,
> >>> including stout. CMake 3 has a notion of "interface libraries" where a
> >>> header-only library can be represented as a CMake target like any other
> >>> library. With this done correctly, linking stout to boost is as simple
> as
> >>> `target_link_libraries(stout boost)`, and linking libprocess to stout
> is
> >>> `target_link_libraries(process stout)`, and the boost dependency is
> >>> understood transitively.
> >>>
> >>> The second phase was refactoring the Mesos build itself (that is, not
> the
> >>> third party dependencies). With all our dependencies imported
> properly, I
> >>> was able to delete the files of extraneous information such as
> >>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This information
> is
> >>> instead correctly stored in the aforementioned CMake dependency graph.
> >>>
> >>> With this all put together, the entirety of the required CMake code to
> >>> build the agent executable, on all platforms, is the following _two
> >>> lines_:
> >>>
> >>>     add_executable(mesos-agent main.cpp)
> >>>     target_link_libraries(mesos-agent PRIVATE mesos)
> >>>
> >>> All necessary link and compilation flags are parsed by CMake through
> the
> >>> dependency graph as it visits libmesos as the mesos-agent dependency.
> >>>
> >>> I keep an updated tree on GitHub here for easy testing:
> >>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
> >>> ave/CMakeLists.txt
> >>>
> >>> As we move closer to deprecating Autotools, I wanted to ensure that the
> >>> replacement build system was as correct and easy-to-use as possible. As
> >>> with any build system (and any software engineering effort), there is
> >>> always more to clean up and improve. However, I am satisfied with the
> >>> result of my efforts, and I hope this build system makes your work as
> >>> developers easier. If you have the time, please take a look at the the
> >>> patches and give it a test. Let me know know I can make it work better
> >>> for
> >>> you.
> >>>
> >>> Thank you,
> >>>
> >>> Andrew Schwartzmeyer
> >>>
> >>>
>



-- 
Cheers,

Zhitao Li

Re: CMake build refactoring

Posted by Vinod Kone <vi...@apache.org>.
+1

On Fri, Aug 25, 2017 at 12:08 PM, <an...@schwartzmeyer.com> wrote:

> I have the Java libraries building now on both Linux and Windows (though
> the initial setup on Windows is a bit annoying to get all the tools in the
> right place, I'd like to follow up this chain with a `docs/cmake.md` with
> all the odd little bits of information).
>
> I also enabled the ZooKeeper unit tests (which I needed in order to
> integration test the patches that went upstream to ZooKeeper to give their
> C Client library a CMake build, too).
>
> Furthermore, with the Java library, I've successfully run Marathon against
> CMake built Mesos libraries on Linux.
>
> I do not have the Python libraries building yet, and there is a "Java
> examples" library that is not yet built. I'll make sure to file issues to
> update these.
>
> Joe is working to update the Mesosphere CI, I will check with him about
> updating `docker-build.sh`.
>
> Thanks Vinod!
>
>
> On 08/25/2017 11:59 am, Vinod Kone wrote:
>
>> This is great to hear. Thanks for your perseverance on making this happen
>> Andy!
>>
>> Btw, do we build Java and Python libraries too now?
>>
>> Once we land the CMake changes, lets make sure to update
>> `support/docker-build.sh` to make sure it is continuously being tested on
>> our CI.
>>
>> On Fri, Aug 25, 2017 at 11:47 AM, <an...@schwartzmeyer.com> wrote:
>>
>> Hello Mesos developers,
>>>
>>> I wanted to take the time to announce a (long) patch chain that fixes up
>>> the CMake build system. There were a lot of open bugs due to the initial
>>> design (and constraints of CMake 2), and by moving to CMake 3, I was able
>>> to rewrite the CMake build in a target-based dependency graph manner (the
>>> way CMake is meant to be used).
>>>
>>> The very end of the chain is here: https://reviews.apache.org/r/61753/
>>>
>>> Joseph Wu is currently shepherding it. That final patch includes the what
>>> and the why explanation of this patch series.
>>>
>>> My main guideline in this effort was MESOS-3576, which pointed out the
>>> incorrectness of the link flags generated by CMake. This has been
>>> resolved,
>>> and moreover, a significant amount of superfluous CMake code was deleted
>>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>>>
>>> I took a two-phase approach to this reworking. The first phase imported
>>> all of our third party dependencies correctly. Instead of linking to
>>> these
>>> dependencies by manually including their various include/library
>>> directories and compilation/link flags, each dependency was added to the
>>> CMake graph as an imported target. All necessary information for linking
>>> to
>>> the dependency is recorded in its target, and used is transitively passed
>>> to targets which take it as dependency by CMake. Adding a new dependency
>>> is
>>> now relatively simple, with many easy-to-replicate patterns available in
>>> 3rdparty/CMakeLists.txt.
>>>
>>> An important note is that this was also done for header-only libraries,
>>> including stout. CMake 3 has a notion of "interface libraries" where a
>>> header-only library can be represented as a CMake target like any other
>>> library. With this done correctly, linking stout to boost is as simple as
>>> `target_link_libraries(stout boost)`, and linking libprocess to stout is
>>> `target_link_libraries(process stout)`, and the boost dependency is
>>> understood transitively.
>>>
>>> The second phase was refactoring the Mesos build itself (that is, not the
>>> third party dependencies). With all our dependencies imported properly, I
>>> was able to delete the files of extraneous information such as
>>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This information is
>>> instead correctly stored in the aforementioned CMake dependency graph.
>>>
>>> With this all put together, the entirety of the required CMake code to
>>> build the agent executable, on all platforms, is the following _two
>>> lines_:
>>>
>>>     add_executable(mesos-agent main.cpp)
>>>     target_link_libraries(mesos-agent PRIVATE mesos)
>>>
>>> All necessary link and compilation flags are parsed by CMake through the
>>> dependency graph as it visits libmesos as the mesos-agent dependency.
>>>
>>> I keep an updated tree on GitHub here for easy testing:
>>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
>>> ave/CMakeLists.txt
>>>
>>> As we move closer to deprecating Autotools, I wanted to ensure that the
>>> replacement build system was as correct and easy-to-use as possible. As
>>> with any build system (and any software engineering effort), there is
>>> always more to clean up and improve. However, I am satisfied with the
>>> result of my efforts, and I hope this build system makes your work as
>>> developers easier. If you have the time, please take a look at the the
>>> patches and give it a test. Let me know know I can make it work better
>>> for
>>> you.
>>>
>>> Thank you,
>>>
>>> Andrew Schwartzmeyer
>>>
>>>

Re: CMake build refactoring

Posted by an...@schwartzmeyer.com.
I have the Java libraries building now on both Linux and Windows (though 
the initial setup on Windows is a bit annoying to get all the tools in 
the right place, I'd like to follow up this chain with a `docs/cmake.md` 
with all the odd little bits of information).

I also enabled the ZooKeeper unit tests (which I needed in order to 
integration test the patches that went upstream to ZooKeeper to give 
their C Client library a CMake build, too).

Furthermore, with the Java library, I've successfully run Marathon 
against CMake built Mesos libraries on Linux.

I do not have the Python libraries building yet, and there is a "Java 
examples" library that is not yet built. I'll make sure to file issues 
to update these.

Joe is working to update the Mesosphere CI, I will check with him about 
updating `docker-build.sh`.

Thanks Vinod!

On 08/25/2017 11:59 am, Vinod Kone wrote:
> This is great to hear. Thanks for your perseverance on making this 
> happen
> Andy!
> 
> Btw, do we build Java and Python libraries too now?
> 
> Once we land the CMake changes, lets make sure to update
> `support/docker-build.sh` to make sure it is continuously being tested 
> on
> our CI.
> 
> On Fri, Aug 25, 2017 at 11:47 AM, <an...@schwartzmeyer.com> wrote:
> 
>> Hello Mesos developers,
>> 
>> I wanted to take the time to announce a (long) patch chain that fixes 
>> up
>> the CMake build system. There were a lot of open bugs due to the 
>> initial
>> design (and constraints of CMake 2), and by moving to CMake 3, I was 
>> able
>> to rewrite the CMake build in a target-based dependency graph manner 
>> (the
>> way CMake is meant to be used).
>> 
>> The very end of the chain is here: https://reviews.apache.org/r/61753/
>> 
>> Joseph Wu is currently shepherding it. That final patch includes the 
>> what
>> and the why explanation of this patch series.
>> 
>> My main guideline in this effort was MESOS-3576, which pointed out the
>> incorrectness of the link flags generated by CMake. This has been 
>> resolved,
>> and moreover, a significant amount of superfluous CMake code was 
>> deleted
>> (that final patch was about a 4 to 1 ratio of deletions to adds).
>> 
>> I took a two-phase approach to this reworking. The first phase 
>> imported
>> all of our third party dependencies correctly. Instead of linking to 
>> these
>> dependencies by manually including their various include/library
>> directories and compilation/link flags, each dependency was added to 
>> the
>> CMake graph as an imported target. All necessary information for 
>> linking to
>> the dependency is recorded in its target, and used is transitively 
>> passed
>> to targets which take it as dependency by CMake. Adding a new 
>> dependency is
>> now relatively simple, with many easy-to-replicate patterns available 
>> in
>> 3rdparty/CMakeLists.txt.
>> 
>> An important note is that this was also done for header-only 
>> libraries,
>> including stout. CMake 3 has a notion of "interface libraries" where a
>> header-only library can be represented as a CMake target like any 
>> other
>> library. With this done correctly, linking stout to boost is as simple 
>> as
>> `target_link_libraries(stout boost)`, and linking libprocess to stout 
>> is
>> `target_link_libraries(process stout)`, and the boost dependency is
>> understood transitively.
>> 
>> The second phase was refactoring the Mesos build itself (that is, not 
>> the
>> third party dependencies). With all our dependencies imported 
>> properly, I
>> was able to delete the files of extraneous information such as
>> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This information 
>> is
>> instead correctly stored in the aforementioned CMake dependency graph.
>> 
>> With this all put together, the entirety of the required CMake code to
>> build the agent executable, on all platforms, is the following _two 
>> lines_:
>> 
>>     add_executable(mesos-agent main.cpp)
>>     target_link_libraries(mesos-agent PRIVATE mesos)
>> 
>> All necessary link and compilation flags are parsed by CMake through 
>> the
>> dependency graph as it visits libmesos as the mesos-agent dependency.
>> 
>> I keep an updated tree on GitHub here for easy testing:
>> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
>> ave/CMakeLists.txt
>> 
>> As we move closer to deprecating Autotools, I wanted to ensure that 
>> the
>> replacement build system was as correct and easy-to-use as possible. 
>> As
>> with any build system (and any software engineering effort), there is
>> always more to clean up and improve. However, I am satisfied with the
>> result of my efforts, and I hope this build system makes your work as
>> developers easier. If you have the time, please take a look at the the
>> patches and give it a test. Let me know know I can make it work better 
>> for
>> you.
>> 
>> Thank you,
>> 
>> Andrew Schwartzmeyer
>> 

Re: CMake build refactoring

Posted by Vinod Kone <vi...@apache.org>.
This is great to hear. Thanks for your perseverance on making this happen
Andy!

Btw, do we build Java and Python libraries too now?

Once we land the CMake changes, lets make sure to update
`support/docker-build.sh` to make sure it is continuously being tested on
our CI.

On Fri, Aug 25, 2017 at 11:47 AM, <an...@schwartzmeyer.com> wrote:

> Hello Mesos developers,
>
> I wanted to take the time to announce a (long) patch chain that fixes up
> the CMake build system. There were a lot of open bugs due to the initial
> design (and constraints of CMake 2), and by moving to CMake 3, I was able
> to rewrite the CMake build in a target-based dependency graph manner (the
> way CMake is meant to be used).
>
> The very end of the chain is here: https://reviews.apache.org/r/61753/
>
> Joseph Wu is currently shepherding it. That final patch includes the what
> and the why explanation of this patch series.
>
> My main guideline in this effort was MESOS-3576, which pointed out the
> incorrectness of the link flags generated by CMake. This has been resolved,
> and moreover, a significant amount of superfluous CMake code was deleted
> (that final patch was about a 4 to 1 ratio of deletions to adds).
>
> I took a two-phase approach to this reworking. The first phase imported
> all of our third party dependencies correctly. Instead of linking to these
> dependencies by manually including their various include/library
> directories and compilation/link flags, each dependency was added to the
> CMake graph as an imported target. All necessary information for linking to
> the dependency is recorded in its target, and used is transitively passed
> to targets which take it as dependency by CMake. Adding a new dependency is
> now relatively simple, with many easy-to-replicate patterns available in
> 3rdparty/CMakeLists.txt.
>
> An important note is that this was also done for header-only libraries,
> including stout. CMake 3 has a notion of "interface libraries" where a
> header-only library can be represented as a CMake target like any other
> library. With this done correctly, linking stout to boost is as simple as
> `target_link_libraries(stout boost)`, and linking libprocess to stout is
> `target_link_libraries(process stout)`, and the boost dependency is
> understood transitively.
>
> The second phase was refactoring the Mesos build itself (that is, not the
> third party dependencies). With all our dependencies imported properly, I
> was able to delete the files of extraneous information such as
> `MasterConfigure.cmake`, `AgentConfigure.cmake`, etc. This information is
> instead correctly stored in the aforementioned CMake dependency graph.
>
> With this all put together, the entirety of the required CMake code to
> build the agent executable, on all platforms, is the following _two lines_:
>
>     add_executable(mesos-agent main.cpp)
>     target_link_libraries(mesos-agent PRIVATE mesos)
>
> All necessary link and compilation flags are parsed by CMake through the
> dependency graph as it visits libmesos as the mesos-agent dependency.
>
> I keep an updated tree on GitHub here for easy testing:
> https://github.com/andschwa/mesos/blob/cmake-refactor/src/sl
> ave/CMakeLists.txt
>
> As we move closer to deprecating Autotools, I wanted to ensure that the
> replacement build system was as correct and easy-to-use as possible. As
> with any build system (and any software engineering effort), there is
> always more to clean up and improve. However, I am satisfied with the
> result of my efforts, and I hope this build system makes your work as
> developers easier. If you have the time, please take a look at the the
> patches and give it a test. Let me know know I can make it work better for
> you.
>
> Thank you,
>
> Andrew Schwartzmeyer
>