Reorganize 3rdparty directory

[Kapil Arya <ka...@mesosphere.io>]

Hi All,

TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into 3rdparty/.
(Optionally) Move libprocess/stout to the top-level directory.

I wanted to start some discussion around reorganizing stuff inside
"3rdparty". I apologize for the length of the email, please bear with me.

Traditionally, 3rdparty has been used to hold all Mesos dependencies
(zookeeper, libprocess, protobuf, stout, etc.). Further,
libprocess/3rdparty was to hold all libprocess dependencies (which may in
turn be Mesos dependencies as well).

As I understand, the original motivation was to emphasize that libprocess
is an independent project which depends on "stout", which in turn is also
an independent project.

However, in the current code base, we don't strictly follow the 3rdparty
structure. For example, stout has a dependency on picojson and
google-protobuf, but we don't put these two packages inside
3rdparty/libprocess/3rdparty/stout/3rdparty/.

In light of these anomalies, I want to propose that we flatten out the
3rdparty directory and put all packages (libprocess, stout, protobuf,
picojson, zookeeper, etc.) at the same level in 3rdparty. We can still use
"--with-XYZ=..." to the full extent as needed.

To take it a step further, I want to propose that we bring libprocess and
stout out of 3rdparty/ and move them at the top level (i.e., make them
peers of src/). That way, all code in 3rdparty/ is stuff from "third"
parties and is used only when "--with-bundled" is defined (by default).
This hierarchy will still allow us to keep libprocess and stout as separate
independent projects.

The motivation for this proposal came when dealing with 3rdparty
dependencies for module development. A module developer needs access to
protobuf, picojson, glog, etc., and for that matter, the exact versions of
these packages that Mesos was compiled with.

We want to solve this problem by installing module-specific 3rdparty
dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
configured with something like "--enable-install-module-dependencies").
(There is a discussion going on in a separate thread).

Further, as of today, when we install Mesos using "make install", we
install stout headers in "${prefix}/include/stout". However, stout has
dependencies on picojson[1] and google-protobuf headers which may not be
present on the machine. This leaves stout, and in turn libprocess and Mesos
headers, fairly broken. I understand that this issue is somewhat orthogonal
to the main issue being discussed in this mail, but I wanted to put it out
since it's related.

Any thoughts, comments, concerns are most welcome!

Best,
Kapil


[1]: Picojson issue was resolved as part of
https://reviews.apache.org/r/41424/ which installs picojson.h into the
include-dir.

Re: Reorganize 3rdparty directory

[Kapil Arya <ka...@mesosphere.io>]

Hi All,

Sorry for the delay in getting back to this issue. I have now created RRs
starting at https://reviews.apache.org/r/46514/ that flatten 3rdparty/. We
have also added a new configure flag --enable-install-module-dependencies,
to allow installation of module dependencies.

Over the next few days, we'll get it reviewed and hopefully commit it to
make it into 0.29.0.

Alex,  would you have some cycles to rebase your CMake POC to this review
chain?

Best,
Kapil

On Tue, Mar 1, 2016 at 5:35 AM, Alex Clemmer <cl...@gmail.com>
wrote:

> Good question. At this point, libmesos should build, but the CMake
> build has fallen a bit behind, since I've been mostly concentrating on
> the integration work. That said, I do expect a bunch of patches to go
> in tomorrow that will re-un-break the libmesos CMake build, and
> following that, the patches that unbreak the agent and master should
> follow, maybe in the next week, depending on the committer bandwidth.
>
> On Fri, Feb 26, 2016 at 10:07 PM, Marco Massenzio <m....@gmail.com>
> wrote:
> >>
> >> Overall, the changes are extremely small. We (apparently) did a pretty
> >> good job of making the CMake configuration scripts modular and
> >> consumable by arbitrary projects
> >
> >
> > ah, the warm, fuzzy feeling that one experiences when all the hard work
> of
> > abstracting stuff pays off :-)
> > well done, Alex!
> >
> > A quick question for you (as you recall, I have great stakes in - but
> > minimal understanding of - cmake) - which targets should be expected to
> > work for Cmake?
> > I had to add a few (minor) fixes to make CLion fully understand Mesos (it
> > still has a few "bogus" errors, but it's by and large, greatly usable -
> and
> > beats Eclipse CDT any day).
> >
> > Also, please let me know whether there's anything you'd like me to try
> out
> > and report back.
> >
> > Thanks!
> >
> > --
> > *Marco Massenzio*
> > http://codetrips.com
> >
> > On Fri, Feb 26, 2016 at 11:44 AM, Alex Clemmer <
> clemmer.alexander@gmail.com>
> > wrote:
> >
> >> Ah, I see now that I coudl have done better by splitting the patch
> >> into two patches -- one where we move everything, and one where we
> >> change the cmake files.
> >>
> >> THe important parts of the patch are:
> >>
> >> * Adding the contents of `3rdparty/libprocess/3rdparty/CMakeLists.txt`
> >> -> `3rdparty/CMakeLists.txt`; makes since, because we're merging the
> >> two 3rdparty folders, so we only need one CMakeLists.txt. We just dump
> >> the contents unchanged into the other one.
> >> * `3rdparty/libprocess/cmake/Process3rdpartyConfigure.cmake` changing
> some
> >> paths
> >> * `3rdparty/libprocess/CMakeLists.txt` moving a call to `include`.
> >>
> >> And that's about it. Everything else is just moving.
> >>
> >> On Fri, Feb 26, 2016 at 11:36 AM, Alex Clemmer
> >> <cl...@gmail.com> wrote:
> >> > Folks:
> >> >
> >> > Took about 30 minutes to put together a prototype of the great
> >> > `3rdparty` flattening. Please see the (extremely preliminary)
> >> > review[1] or my working branch[2] for a really close approximation of
> >> > the number of changes to the CMake build system we'd need to support
> >> > this flattening.
> >> >
> >> > Overall, the changes are extremely small. We (apparently) did a pretty
> >> > good job of making the CMake configuration scripts modular and
> >> > consumable by arbitrary projects, so the changes are mostly things
> >> > like "move the config `include` call over there instead of being over
> >> > there" and "change a handful of path variables to reflect the new dir
> >> > structure". (I hope, btw, that it doesn't seem arrogant to say this,
> >> > but I think it is justified given the relatively minor changes in the
> >> > actual CMake code.)
> >> >
> >> > Feel free to remix/take/throw away any subset of this. I don't need
> >> > any credit for it in the final patch if someone marches off in that
> >> > direction. :)
> >> >
> >> > [1] https://reviews.apache.org/r/44099/
> >> > [2]
> https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd
> >> >
> >> > On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <kl...@gmail.com>
> wrote:
> >> >> I am also a fan of git submodules in the long term, but avoiding them
> >> >> in the short term.  We should get things organized as we want them
> >> >> first, and then start thinking about pulling libprocess/stout out
> into
> >> >> submodules later (while also preserving their history!)
> >> >>
> >> >> I disagree with moving libprocess and stout up to the same level as
> >> >> src/. If we want to make sure they don't bleed into Mesos proper, we
> >> >> really should treat them the same as any other 3rdparty code that we
> >> >> depend on.  This will become more relevant when/if we move them into
> >> >> submodules.
> >> >>
> >> >> Given all that, the only real challenge with flattening our 3rdparty
> >> >> dependencies into a single folder should be the changes we have to
> >> >> make to our configure.ac and Makefile.am scripts to know where to
> look
> >> >> for their dependencies now.  In the end these should be URLs to
> >> >> versioned tarballs that we host somewhere (or git repos that we can
> >> >> have forked and tagged with specific versions).  In the short term
> >> >> these can just be relative paths in the mesos tree though.
> >> >>
> >> >> On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io>
> >> wrote:
> >> >>> Thanks for bringing it up Alexander!
> >> >>>
> >> >>> I don't have a strong opinion wrt git submodules since I don't have
> >> >>> much experience with them personally. Having said that, I would like
> >> >>> to go conservative on this one (baby steps :-) ).
> >> >>>
> >> >>> Further, I do understand that moving libprocess and stout
> directories
> >> >>> will be painful for people who already have several branches and
> will
> >> >>> have conflicts. But I do think, there are some interim solutions as
> >> >>> well (for example, move libprocess/stout to wherever we want, but
> keep
> >> >>> a symlink from 3rdparty/libprocess, etc, to those new locations for
> >> >>> some time). I am sure there are better solutions out there, but this
> >> >>> should work too.
> >> >>>
> >> >>> Best,
> >> >>> Kapil
> >> >>>
> >> >>> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers
> >> >>> <ew...@groupon.com.invalid> wrote:
> >> >>>> If we go to git submodules, please ensure there are good docs
> around
> >> how to
> >> >>>> update cloned repos.
> >> >>>>
> >> >>>> e.g., From ansible:
> >> https://docs.ansible.com/ansible/intro_installation.html
> >> >>>>
> >> >>>> Note when updating ansible, be sure to not only update the source
> >> tree, but
> >> >>>> also the “submodules” in git which point at Ansible’s own modules
> >> (not the
> >> >>>> same kind of modules, alas).
> >> >>>>
> >> >>>> $ git pull --rebase
> >> >>>> $ git submodule update --init --recursive
> >> >>>>
> >> >>>> Thanks,
> >> >>>>
> >> >>>> - Erik
> >> >>>>
> >> >>>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <
> >> alexander@mesosphere.io>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> +1
> >> >>>>> I am one who is totally in for that change. It is not only the
> >> directories
> >> >>>>> problem, but the structure which has led that the stout tests
> (which
> >> do
> >> >>>>> need to be compiled) are actually managed in the libprocess
> >> Makefile, on
> >> >>>>> top of all the things you have already mentioned.
> >> >>>>>
> >> >>>>>
> >> >>>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io>
> wrote:
> >> >>>>> >
> >> >>>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com>
> >> wrote:
> >> >>>>> >> Kapil,
> >> >>>>> >>
> >> >>>>> >> I guess what I want to understand is why the existing structure
> >> makes it
> >> >>>>> >> hard for you to do the things that you want to do (installing
> >> >>>>> >> module-specific 3rdparty dependencies into
> >> "${pkglibdir}/3rdparty" as
> >> >>>>> part
> >> >>>>> >> of "make install").
> >> >>>>> >
> >> >>>>> > Let me see if I can answer that :-).
> >> >>>>> >
> >> >>>>> > This is somewhat related. For example, if we want to install
> >> protobuf
> >> >>>>> > in 3rdparty/{include,lib} (for module developers to use them
> >> without
> >> >>>>> > doing a proper mesos installation), you need to provide the
> correct
> >> >>>>> > "--prefix" flag that points to 3rdparty/. However, due to
> multiple
> >> >>>>> > levels of configure.ac, the "--prefix" can at best be
> generated by
> >> >>>>> > prepending "../../../" to get to the great-grandparent
> directory.
> >> This
> >> >>>>> > is because we have a separate configure.ac which manages
> >> >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around
> it,
> >> >>>>> > but they are not clean.
> >> >>>>> >
> >> >>>>> > Similar thing holds for system-wide installation of these
> 3rdparty
> >> >>>>> > packages. For example, ideally, we would want to use
> >> >>>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However,
> >> since
> >> >>>>> > they are part of libprocess package, we don't get the correct
> >> >>>>> > directory and have to use either hardwired $pkglibdir, or
> somehow
> >> pass
> >> >>>>> > it from the top-level configure all the way down to
> >> >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >> The only reason you mentioned in the original email is that "in
> >> the
> >> >>>>> current
> >> >>>>> >> code base, we don't strictly follow the 3rdparty structure",
> >> which IMO
> >> >>>>> is
> >> >>>>> >> not a very convincing reason for such a big change.
> >> >>>>> >
> >> >>>>> > How about a not so big change? :-). What if we just move
> >> >>>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while
> leaving
> >> >>>>> > stout as is? That is not a big change since we are not touching
> >> >>>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure
> it
> >> >>>>> > will be cleaner and simpler than what we have right now.
> >> >>>>> >
> >> >>>>> > As a later time, we can then consider moving stout out to
> 3rdparty/
> >> >>>>> > while leaving libprocess as is. But that's something we can
> decide
> >> >>>>> > later and leave stout as an exception for now.
> >> >>>>> >
> >> >>>>> > BTW, if we were to install all the 3rdparty packages in
> 3rdparty/,
> >> >>>>> > that would also cut down a lot on the compiler flags (i.e.,
> fewer
> >> "-I"
> >> >>>>> > and "-L" flags) :-).
> >> >>>>> >
> >> >>>>> > Kapil
> >> >>>>> >
> >> >>>>> >>
> >> >>>>> >> - Jie
> >> >>>>> >>
> >> >>>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <
> kapil@mesosphere.io>
> >> wrote:
> >> >>>>> >>
> >> >>>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com>
> >> wrote:
> >> >>>>> >>>>
> >> >>>>> >>>>>
> >> >>>>> >>>>> However, in the current code base, we don't strictly follow
> the
> >> >>>>> >>> 3rdparty
> >> >>>>> >>>>> structure. For example, stout has a dependency on picojson
> and
> >> >>>>> >>>>> google-protobuf, but we don't put these two packages inside
> >> >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >> >>>>> >>>>
> >> >>>>> >>>>
> >> >>>>> >>>> My understanding is that stout is header only. So it does not
> >> have to
> >> >>>>> >>>> bundle 3rdparty libraries. The user of stout is responsible
> for
> >> >>>>> bundling
> >> >>>>> >>>> them if they are used.
> >> >>>>> >>>
> >> >>>>> >>>
> >> >>>>> >>> I don't think being header-only is an excuse to have a broken
> >> >>>>> >>> installation :-). Further, we don't make it easier for the
> user
> >> to get
> >> >>>>> >>> the 3rdparty binaries either. For example, if the user has a
> >> different
> >> >>>>> >>> version of protobuf installed on the system, the compilation
> of
> >> any
> >> >>>>> >>> program that uses stout will fail spectacularly!
> >> >>>>> >>>
> >> >>>>> >>> Having said that, the gist here is that we have somewhat
> >> deviated from
> >> >>>>> >>> original motivation behind the 3rdparty directory and it would
> >> be nice
> >> >>>>> >>> if we can have a flatter structure.
> >> >>>>> >>>
> >> >>>>> >>>>
> >> >>>>> >>>>
> >> >>>>> >>>> - Jie
> >> >>>>> >>>>
> >> >>>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <
> kapil@mesosphere.io
> >> >
> >> >>>>> wrote:
> >> >>>>> >>>>
> >> >>>>> >>>>> Hi All,
> >> >>>>> >>>>>
> >> >>>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/*
> into
> >> >>>>> >>> 3rdparty/.
> >> >>>>> >>>>> (Optionally) Move libprocess/stout to the top-level
> directory.
> >> >>>>> >>>>>
> >> >>>>> >>>>> I wanted to start some discussion around reorganizing stuff
> >> inside
> >> >>>>> >>>>> "3rdparty". I apologize for the length of the email, please
> >> bear with
> >> >>>>> >>> me.
> >> >>>>> >>>>>
> >> >>>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos
> >> dependencies
> >> >>>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
> >> >>>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies
> >> (which
> >> >>>>> may
> >> >>>>> >>> in
> >> >>>>> >>>>> turn be Mesos dependencies as well).
> >> >>>>> >>>>>
> >> >>>>> >>>>> As I understand, the original motivation was to emphasize
> that
> >> >>>>> >>> libprocess
> >> >>>>> >>>>> is an independent project which depends on "stout", which in
> >> turn is
> >> >>>>> >>> also
> >> >>>>> >>>>> an independent project.
> >> >>>>> >>>>>
> >> >>>>> >>>>> However, in the current code base, we don't strictly follow
> the
> >> >>>>> >>> 3rdparty
> >> >>>>> >>>>> structure. For example, stout has a dependency on picojson
> and
> >> >>>>> >>>>> google-protobuf, but we don't put these two packages inside
> >> >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >> >>>>> >>>>>
> >> >>>>> >>>>> In light of these anomalies, I want to propose that we
> flatten
> >> out
> >> >>>>> the
> >> >>>>> >>>>> 3rdparty directory and put all packages (libprocess, stout,
> >> protobuf,
> >> >>>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We
> >> can
> >> >>>>> still
> >> >>>>> >>> use
> >> >>>>> >>>>> "--with-XYZ=..." to the full extent as needed.
> >> >>>>> >>>>>
> >> >>>>> >>>>> To take it a step further, I want to propose that we bring
> >> libprocess
> >> >>>>> >>> and
> >> >>>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e.,
> >> make
> >> >>>>> them
> >> >>>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff
> from
> >> "third"
> >> >>>>> >>>>> parties and is used only when "--with-bundled" is defined
> (by
> >> >>>>> default).
> >> >>>>> >>>>> This hierarchy will still allow us to keep libprocess and
> >> stout as
> >> >>>>> >>> separate
> >> >>>>> >>>>> independent projects.
> >> >>>>> >>>>>
> >> >>>>> >>>>> The motivation for this proposal came when dealing with
> >> 3rdparty
> >> >>>>> >>>>> dependencies for module development. A module developer
> needs
> >> access
> >> >>>>> to
> >> >>>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the
> exact
> >> >>>>> >>> versions of
> >> >>>>> >>>>> these packages that Mesos was compiled with.
> >> >>>>> >>>>>
> >> >>>>> >>>>> We want to solve this problem by installing module-specific
> >> 3rdparty
> >> >>>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make
> >> install"
> >> >>>>> (if
> >> >>>>> >>>>> configured with something like
> >> >>>>> "--enable-install-module-dependencies").
> >> >>>>> >>>>> (There is a discussion going on in a separate thread).
> >> >>>>> >>>>>
> >> >>>>> >>>>> Further, as of today, when we install Mesos using "make
> >> install", we
> >> >>>>> >>>>> install stout headers in "${prefix}/include/stout". However,
> >> stout
> >> >>>>> has
> >> >>>>> >>>>> dependencies on picojson[1] and google-protobuf headers
> which
> >> may not
> >> >>>>> >>> be
> >> >>>>> >>>>> present on the machine. This leaves stout, and in turn
> >> libprocess and
> >> >>>>> >>> Mesos
> >> >>>>> >>>>> headers, fairly broken. I understand that this issue is
> >> somewhat
> >> >>>>> >>> orthogonal
> >> >>>>> >>>>> to the main issue being discussed in this mail, but I wanted
> >> to put
> >> >>>>> it
> >> >>>>> >>> out
> >> >>>>> >>>>> since it's related.
> >> >>>>> >>>>>
> >> >>>>> >>>>> Any thoughts, comments, concerns are most welcome!
> >> >>>>> >>>>>
> >> >>>>> >>>>> Best,
> >> >>>>> >>>>> Kapil
> >> >>>>> >>>>>
> >> >>>>> >>>>>
> >> >>>>> >>>>> [1]: Picojson issue was resolved as part of
> >> >>>>> >>>>> https://reviews.apache.org/r/41424/ which installs
> picojson.h
> >> into
> >> >>>>> the
> >> >>>>> >>>>> include-dir.
> >> >>>>> >>>>>
> >> >>>>> >>>
> >> >>>>>
> >> >>>>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> ~Kevin
> >> >
> >> >
> >> >
> >> > --
> >> > Alex
> >> >
> >> > Theory is the first term in the Taylor series of practice. -- Thomas M
> >> > Cover (1992)
> >>
> >>
> >>
> >> --
> >> Alex
> >>
> >> Theory is the first term in the Taylor series of practice. -- Thomas M
> >> Cover (1992)
> >>
>
>
>
> --
> Alex
>
> Theory is the first term in the Taylor series of practice. -- Thomas M
> Cover (1992)
>

Re: Reorganize 3rdparty directory

[Alex Clemmer <cl...@gmail.com>]

Good question. At this point, libmesos should build, but the CMake
build has fallen a bit behind, since I've been mostly concentrating on
the integration work. That said, I do expect a bunch of patches to go
in tomorrow that will re-un-break the libmesos CMake build, and
following that, the patches that unbreak the agent and master should
follow, maybe in the next week, depending on the committer bandwidth.

On Fri, Feb 26, 2016 at 10:07 PM, Marco Massenzio <m....@gmail.com> wrote:
>>
>> Overall, the changes are extremely small. We (apparently) did a pretty
>> good job of making the CMake configuration scripts modular and
>> consumable by arbitrary projects
>
>
> ah, the warm, fuzzy feeling that one experiences when all the hard work of
> abstracting stuff pays off :-)
> well done, Alex!
>
> A quick question for you (as you recall, I have great stakes in - but
> minimal understanding of - cmake) - which targets should be expected to
> work for Cmake?
> I had to add a few (minor) fixes to make CLion fully understand Mesos (it
> still has a few "bogus" errors, but it's by and large, greatly usable - and
> beats Eclipse CDT any day).
>
> Also, please let me know whether there's anything you'd like me to try out
> and report back.
>
> Thanks!
>
> --
> *Marco Massenzio*
> http://codetrips.com
>
> On Fri, Feb 26, 2016 at 11:44 AM, Alex Clemmer <cl...@gmail.com>
> wrote:
>
>> Ah, I see now that I coudl have done better by splitting the patch
>> into two patches -- one where we move everything, and one where we
>> change the cmake files.
>>
>> THe important parts of the patch are:
>>
>> * Adding the contents of `3rdparty/libprocess/3rdparty/CMakeLists.txt`
>> -> `3rdparty/CMakeLists.txt`; makes since, because we're merging the
>> two 3rdparty folders, so we only need one CMakeLists.txt. We just dump
>> the contents unchanged into the other one.
>> * `3rdparty/libprocess/cmake/Process3rdpartyConfigure.cmake` changing some
>> paths
>> * `3rdparty/libprocess/CMakeLists.txt` moving a call to `include`.
>>
>> And that's about it. Everything else is just moving.
>>
>> On Fri, Feb 26, 2016 at 11:36 AM, Alex Clemmer
>> <cl...@gmail.com> wrote:
>> > Folks:
>> >
>> > Took about 30 minutes to put together a prototype of the great
>> > `3rdparty` flattening. Please see the (extremely preliminary)
>> > review[1] or my working branch[2] for a really close approximation of
>> > the number of changes to the CMake build system we'd need to support
>> > this flattening.
>> >
>> > Overall, the changes are extremely small. We (apparently) did a pretty
>> > good job of making the CMake configuration scripts modular and
>> > consumable by arbitrary projects, so the changes are mostly things
>> > like "move the config `include` call over there instead of being over
>> > there" and "change a handful of path variables to reflect the new dir
>> > structure". (I hope, btw, that it doesn't seem arrogant to say this,
>> > but I think it is justified given the relatively minor changes in the
>> > actual CMake code.)
>> >
>> > Feel free to remix/take/throw away any subset of this. I don't need
>> > any credit for it in the final patch if someone marches off in that
>> > direction. :)
>> >
>> > [1] https://reviews.apache.org/r/44099/
>> > [2] https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd
>> >
>> > On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <kl...@gmail.com> wrote:
>> >> I am also a fan of git submodules in the long term, but avoiding them
>> >> in the short term.  We should get things organized as we want them
>> >> first, and then start thinking about pulling libprocess/stout out into
>> >> submodules later (while also preserving their history!)
>> >>
>> >> I disagree with moving libprocess and stout up to the same level as
>> >> src/. If we want to make sure they don't bleed into Mesos proper, we
>> >> really should treat them the same as any other 3rdparty code that we
>> >> depend on.  This will become more relevant when/if we move them into
>> >> submodules.
>> >>
>> >> Given all that, the only real challenge with flattening our 3rdparty
>> >> dependencies into a single folder should be the changes we have to
>> >> make to our configure.ac and Makefile.am scripts to know where to look
>> >> for their dependencies now.  In the end these should be URLs to
>> >> versioned tarballs that we host somewhere (or git repos that we can
>> >> have forked and tagged with specific versions).  In the short term
>> >> these can just be relative paths in the mesos tree though.
>> >>
>> >> On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io>
>> wrote:
>> >>> Thanks for bringing it up Alexander!
>> >>>
>> >>> I don't have a strong opinion wrt git submodules since I don't have
>> >>> much experience with them personally. Having said that, I would like
>> >>> to go conservative on this one (baby steps :-) ).
>> >>>
>> >>> Further, I do understand that moving libprocess and stout directories
>> >>> will be painful for people who already have several branches and will
>> >>> have conflicts. But I do think, there are some interim solutions as
>> >>> well (for example, move libprocess/stout to wherever we want, but keep
>> >>> a symlink from 3rdparty/libprocess, etc, to those new locations for
>> >>> some time). I am sure there are better solutions out there, but this
>> >>> should work too.
>> >>>
>> >>> Best,
>> >>> Kapil
>> >>>
>> >>> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers
>> >>> <ew...@groupon.com.invalid> wrote:
>> >>>> If we go to git submodules, please ensure there are good docs around
>> how to
>> >>>> update cloned repos.
>> >>>>
>> >>>> e.g., From ansible:
>> https://docs.ansible.com/ansible/intro_installation.html
>> >>>>
>> >>>> Note when updating ansible, be sure to not only update the source
>> tree, but
>> >>>> also the “submodules” in git which point at Ansible’s own modules
>> (not the
>> >>>> same kind of modules, alas).
>> >>>>
>> >>>> $ git pull --rebase
>> >>>> $ git submodule update --init --recursive
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> - Erik
>> >>>>
>> >>>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <
>> alexander@mesosphere.io>
>> >>>> wrote:
>> >>>>
>> >>>>> +1
>> >>>>> I am one who is totally in for that change. It is not only the
>> directories
>> >>>>> problem, but the structure which has led that the stout tests (which
>> do
>> >>>>> need to be compiled) are actually managed in the libprocess
>> Makefile, on
>> >>>>> top of all the things you have already mentioned.
>> >>>>>
>> >>>>>
>> >>>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
>> >>>>> >
>> >>>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com>
>> wrote:
>> >>>>> >> Kapil,
>> >>>>> >>
>> >>>>> >> I guess what I want to understand is why the existing structure
>> makes it
>> >>>>> >> hard for you to do the things that you want to do (installing
>> >>>>> >> module-specific 3rdparty dependencies into
>> "${pkglibdir}/3rdparty" as
>> >>>>> part
>> >>>>> >> of "make install").
>> >>>>> >
>> >>>>> > Let me see if I can answer that :-).
>> >>>>> >
>> >>>>> > This is somewhat related. For example, if we want to install
>> protobuf
>> >>>>> > in 3rdparty/{include,lib} (for module developers to use them
>> without
>> >>>>> > doing a proper mesos installation), you need to provide the correct
>> >>>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple
>> >>>>> > levels of configure.ac, the "--prefix" can at best be generated by
>> >>>>> > prepending "../../../" to get to the great-grandparent directory.
>> This
>> >>>>> > is because we have a separate configure.ac which manages
>> >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
>> >>>>> > but they are not clean.
>> >>>>> >
>> >>>>> > Similar thing holds for system-wide installation of these 3rdparty
>> >>>>> > packages. For example, ideally, we would want to use
>> >>>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However,
>> since
>> >>>>> > they are part of libprocess package, we don't get the correct
>> >>>>> > directory and have to use either hardwired $pkglibdir, or somehow
>> pass
>> >>>>> > it from the top-level configure all the way down to
>> >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
>> >>>>> >
>> >>>>> >
>> >>>>> >> The only reason you mentioned in the original email is that "in
>> the
>> >>>>> current
>> >>>>> >> code base, we don't strictly follow the 3rdparty structure",
>> which IMO
>> >>>>> is
>> >>>>> >> not a very convincing reason for such a big change.
>> >>>>> >
>> >>>>> > How about a not so big change? :-). What if we just move
>> >>>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
>> >>>>> > stout as is? That is not a big change since we are not touching
>> >>>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it
>> >>>>> > will be cleaner and simpler than what we have right now.
>> >>>>> >
>> >>>>> > As a later time, we can then consider moving stout out to 3rdparty/
>> >>>>> > while leaving libprocess as is. But that's something we can decide
>> >>>>> > later and leave stout as an exception for now.
>> >>>>> >
>> >>>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/,
>> >>>>> > that would also cut down a lot on the compiler flags (i.e., fewer
>> "-I"
>> >>>>> > and "-L" flags) :-).
>> >>>>> >
>> >>>>> > Kapil
>> >>>>> >
>> >>>>> >>
>> >>>>> >> - Jie
>> >>>>> >>
>> >>>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io>
>> wrote:
>> >>>>> >>
>> >>>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com>
>> wrote:
>> >>>>> >>>>
>> >>>>> >>>>>
>> >>>>> >>>>> However, in the current code base, we don't strictly follow the
>> >>>>> >>> 3rdparty
>> >>>>> >>>>> structure. For example, stout has a dependency on picojson and
>> >>>>> >>>>> google-protobuf, but we don't put these two packages inside
>> >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>> My understanding is that stout is header only. So it does not
>> have to
>> >>>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for
>> >>>>> bundling
>> >>>>> >>>> them if they are used.
>> >>>>> >>>
>> >>>>> >>>
>> >>>>> >>> I don't think being header-only is an excuse to have a broken
>> >>>>> >>> installation :-). Further, we don't make it easier for the user
>> to get
>> >>>>> >>> the 3rdparty binaries either. For example, if the user has a
>> different
>> >>>>> >>> version of protobuf installed on the system, the compilation of
>> any
>> >>>>> >>> program that uses stout will fail spectacularly!
>> >>>>> >>>
>> >>>>> >>> Having said that, the gist here is that we have somewhat
>> deviated from
>> >>>>> >>> original motivation behind the 3rdparty directory and it would
>> be nice
>> >>>>> >>> if we can have a flatter structure.
>> >>>>> >>>
>> >>>>> >>>>
>> >>>>> >>>>
>> >>>>> >>>> - Jie
>> >>>>> >>>>
>> >>>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <kapil@mesosphere.io
>> >
>> >>>>> wrote:
>> >>>>> >>>>
>> >>>>> >>>>> Hi All,
>> >>>>> >>>>>
>> >>>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>> >>>>> >>> 3rdparty/.
>> >>>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory.
>> >>>>> >>>>>
>> >>>>> >>>>> I wanted to start some discussion around reorganizing stuff
>> inside
>> >>>>> >>>>> "3rdparty". I apologize for the length of the email, please
>> bear with
>> >>>>> >>> me.
>> >>>>> >>>>>
>> >>>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos
>> dependencies
>> >>>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
>> >>>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies
>> (which
>> >>>>> may
>> >>>>> >>> in
>> >>>>> >>>>> turn be Mesos dependencies as well).
>> >>>>> >>>>>
>> >>>>> >>>>> As I understand, the original motivation was to emphasize that
>> >>>>> >>> libprocess
>> >>>>> >>>>> is an independent project which depends on "stout", which in
>> turn is
>> >>>>> >>> also
>> >>>>> >>>>> an independent project.
>> >>>>> >>>>>
>> >>>>> >>>>> However, in the current code base, we don't strictly follow the
>> >>>>> >>> 3rdparty
>> >>>>> >>>>> structure. For example, stout has a dependency on picojson and
>> >>>>> >>>>> google-protobuf, but we don't put these two packages inside
>> >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> >>>>> >>>>>
>> >>>>> >>>>> In light of these anomalies, I want to propose that we flatten
>> out
>> >>>>> the
>> >>>>> >>>>> 3rdparty directory and put all packages (libprocess, stout,
>> protobuf,
>> >>>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We
>> can
>> >>>>> still
>> >>>>> >>> use
>> >>>>> >>>>> "--with-XYZ=..." to the full extent as needed.
>> >>>>> >>>>>
>> >>>>> >>>>> To take it a step further, I want to propose that we bring
>> libprocess
>> >>>>> >>> and
>> >>>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e.,
>> make
>> >>>>> them
>> >>>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from
>> "third"
>> >>>>> >>>>> parties and is used only when "--with-bundled" is defined (by
>> >>>>> default).
>> >>>>> >>>>> This hierarchy will still allow us to keep libprocess and
>> stout as
>> >>>>> >>> separate
>> >>>>> >>>>> independent projects.
>> >>>>> >>>>>
>> >>>>> >>>>> The motivation for this proposal came when dealing with
>> 3rdparty
>> >>>>> >>>>> dependencies for module development. A module developer needs
>> access
>> >>>>> to
>> >>>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact
>> >>>>> >>> versions of
>> >>>>> >>>>> these packages that Mesos was compiled with.
>> >>>>> >>>>>
>> >>>>> >>>>> We want to solve this problem by installing module-specific
>> 3rdparty
>> >>>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make
>> install"
>> >>>>> (if
>> >>>>> >>>>> configured with something like
>> >>>>> "--enable-install-module-dependencies").
>> >>>>> >>>>> (There is a discussion going on in a separate thread).
>> >>>>> >>>>>
>> >>>>> >>>>> Further, as of today, when we install Mesos using "make
>> install", we
>> >>>>> >>>>> install stout headers in "${prefix}/include/stout". However,
>> stout
>> >>>>> has
>> >>>>> >>>>> dependencies on picojson[1] and google-protobuf headers which
>> may not
>> >>>>> >>> be
>> >>>>> >>>>> present on the machine. This leaves stout, and in turn
>> libprocess and
>> >>>>> >>> Mesos
>> >>>>> >>>>> headers, fairly broken. I understand that this issue is
>> somewhat
>> >>>>> >>> orthogonal
>> >>>>> >>>>> to the main issue being discussed in this mail, but I wanted
>> to put
>> >>>>> it
>> >>>>> >>> out
>> >>>>> >>>>> since it's related.
>> >>>>> >>>>>
>> >>>>> >>>>> Any thoughts, comments, concerns are most welcome!
>> >>>>> >>>>>
>> >>>>> >>>>> Best,
>> >>>>> >>>>> Kapil
>> >>>>> >>>>>
>> >>>>> >>>>>
>> >>>>> >>>>> [1]: Picojson issue was resolved as part of
>> >>>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h
>> into
>> >>>>> the
>> >>>>> >>>>> include-dir.
>> >>>>> >>>>>
>> >>>>> >>>
>> >>>>>
>> >>>>>
>> >>
>> >>
>> >>
>> >> --
>> >> ~Kevin
>> >
>> >
>> >
>> > --
>> > Alex
>> >
>> > Theory is the first term in the Taylor series of practice. -- Thomas M
>> > Cover (1992)
>>
>>
>>
>> --
>> Alex
>>
>> Theory is the first term in the Taylor series of practice. -- Thomas M
>> Cover (1992)
>>



-- 
Alex

Theory is the first term in the Taylor series of practice. -- Thomas M
Cover (1992)

Re: Reorganize 3rdparty directory

[Marco Massenzio <m....@gmail.com>]

>
> Overall, the changes are extremely small. We (apparently) did a pretty
> good job of making the CMake configuration scripts modular and
> consumable by arbitrary projects


ah, the warm, fuzzy feeling that one experiences when all the hard work of
abstracting stuff pays off :-)
well done, Alex!

A quick question for you (as you recall, I have great stakes in - but
minimal understanding of - cmake) - which targets should be expected to
work for Cmake?
I had to add a few (minor) fixes to make CLion fully understand Mesos (it
still has a few "bogus" errors, but it's by and large, greatly usable - and
beats Eclipse CDT any day).

Also, please let me know whether there's anything you'd like me to try out
and report back.

Thanks!

-- 
*Marco Massenzio*
http://codetrips.com

On Fri, Feb 26, 2016 at 11:44 AM, Alex Clemmer <cl...@gmail.com>
wrote:

> Ah, I see now that I coudl have done better by splitting the patch
> into two patches -- one where we move everything, and one where we
> change the cmake files.
>
> THe important parts of the patch are:
>
> * Adding the contents of `3rdparty/libprocess/3rdparty/CMakeLists.txt`
> -> `3rdparty/CMakeLists.txt`; makes since, because we're merging the
> two 3rdparty folders, so we only need one CMakeLists.txt. We just dump
> the contents unchanged into the other one.
> * `3rdparty/libprocess/cmake/Process3rdpartyConfigure.cmake` changing some
> paths
> * `3rdparty/libprocess/CMakeLists.txt` moving a call to `include`.
>
> And that's about it. Everything else is just moving.
>
> On Fri, Feb 26, 2016 at 11:36 AM, Alex Clemmer
> <cl...@gmail.com> wrote:
> > Folks:
> >
> > Took about 30 minutes to put together a prototype of the great
> > `3rdparty` flattening. Please see the (extremely preliminary)
> > review[1] or my working branch[2] for a really close approximation of
> > the number of changes to the CMake build system we'd need to support
> > this flattening.
> >
> > Overall, the changes are extremely small. We (apparently) did a pretty
> > good job of making the CMake configuration scripts modular and
> > consumable by arbitrary projects, so the changes are mostly things
> > like "move the config `include` call over there instead of being over
> > there" and "change a handful of path variables to reflect the new dir
> > structure". (I hope, btw, that it doesn't seem arrogant to say this,
> > but I think it is justified given the relatively minor changes in the
> > actual CMake code.)
> >
> > Feel free to remix/take/throw away any subset of this. I don't need
> > any credit for it in the final patch if someone marches off in that
> > direction. :)
> >
> > [1] https://reviews.apache.org/r/44099/
> > [2] https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd
> >
> > On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <kl...@gmail.com> wrote:
> >> I am also a fan of git submodules in the long term, but avoiding them
> >> in the short term.  We should get things organized as we want them
> >> first, and then start thinking about pulling libprocess/stout out into
> >> submodules later (while also preserving their history!)
> >>
> >> I disagree with moving libprocess and stout up to the same level as
> >> src/. If we want to make sure they don't bleed into Mesos proper, we
> >> really should treat them the same as any other 3rdparty code that we
> >> depend on.  This will become more relevant when/if we move them into
> >> submodules.
> >>
> >> Given all that, the only real challenge with flattening our 3rdparty
> >> dependencies into a single folder should be the changes we have to
> >> make to our configure.ac and Makefile.am scripts to know where to look
> >> for their dependencies now.  In the end these should be URLs to
> >> versioned tarballs that we host somewhere (or git repos that we can
> >> have forked and tagged with specific versions).  In the short term
> >> these can just be relative paths in the mesos tree though.
> >>
> >> On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io>
> wrote:
> >>> Thanks for bringing it up Alexander!
> >>>
> >>> I don't have a strong opinion wrt git submodules since I don't have
> >>> much experience with them personally. Having said that, I would like
> >>> to go conservative on this one (baby steps :-) ).
> >>>
> >>> Further, I do understand that moving libprocess and stout directories
> >>> will be painful for people who already have several branches and will
> >>> have conflicts. But I do think, there are some interim solutions as
> >>> well (for example, move libprocess/stout to wherever we want, but keep
> >>> a symlink from 3rdparty/libprocess, etc, to those new locations for
> >>> some time). I am sure there are better solutions out there, but this
> >>> should work too.
> >>>
> >>> Best,
> >>> Kapil
> >>>
> >>> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers
> >>> <ew...@groupon.com.invalid> wrote:
> >>>> If we go to git submodules, please ensure there are good docs around
> how to
> >>>> update cloned repos.
> >>>>
> >>>> e.g., From ansible:
> https://docs.ansible.com/ansible/intro_installation.html
> >>>>
> >>>> Note when updating ansible, be sure to not only update the source
> tree, but
> >>>> also the “submodules” in git which point at Ansible’s own modules
> (not the
> >>>> same kind of modules, alas).
> >>>>
> >>>> $ git pull --rebase
> >>>> $ git submodule update --init --recursive
> >>>>
> >>>> Thanks,
> >>>>
> >>>> - Erik
> >>>>
> >>>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <
> alexander@mesosphere.io>
> >>>> wrote:
> >>>>
> >>>>> +1
> >>>>> I am one who is totally in for that change. It is not only the
> directories
> >>>>> problem, but the structure which has led that the stout tests (which
> do
> >>>>> need to be compiled) are actually managed in the libprocess
> Makefile, on
> >>>>> top of all the things you have already mentioned.
> >>>>>
> >>>>>
> >>>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
> >>>>> >
> >>>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com>
> wrote:
> >>>>> >> Kapil,
> >>>>> >>
> >>>>> >> I guess what I want to understand is why the existing structure
> makes it
> >>>>> >> hard for you to do the things that you want to do (installing
> >>>>> >> module-specific 3rdparty dependencies into
> "${pkglibdir}/3rdparty" as
> >>>>> part
> >>>>> >> of "make install").
> >>>>> >
> >>>>> > Let me see if I can answer that :-).
> >>>>> >
> >>>>> > This is somewhat related. For example, if we want to install
> protobuf
> >>>>> > in 3rdparty/{include,lib} (for module developers to use them
> without
> >>>>> > doing a proper mesos installation), you need to provide the correct
> >>>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple
> >>>>> > levels of configure.ac, the "--prefix" can at best be generated by
> >>>>> > prepending "../../../" to get to the great-grandparent directory.
> This
> >>>>> > is because we have a separate configure.ac which manages
> >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
> >>>>> > but they are not clean.
> >>>>> >
> >>>>> > Similar thing holds for system-wide installation of these 3rdparty
> >>>>> > packages. For example, ideally, we would want to use
> >>>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However,
> since
> >>>>> > they are part of libprocess package, we don't get the correct
> >>>>> > directory and have to use either hardwired $pkglibdir, or somehow
> pass
> >>>>> > it from the top-level configure all the way down to
> >>>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
> >>>>> >
> >>>>> >
> >>>>> >> The only reason you mentioned in the original email is that "in
> the
> >>>>> current
> >>>>> >> code base, we don't strictly follow the 3rdparty structure",
> which IMO
> >>>>> is
> >>>>> >> not a very convincing reason for such a big change.
> >>>>> >
> >>>>> > How about a not so big change? :-). What if we just move
> >>>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
> >>>>> > stout as is? That is not a big change since we are not touching
> >>>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it
> >>>>> > will be cleaner and simpler than what we have right now.
> >>>>> >
> >>>>> > As a later time, we can then consider moving stout out to 3rdparty/
> >>>>> > while leaving libprocess as is. But that's something we can decide
> >>>>> > later and leave stout as an exception for now.
> >>>>> >
> >>>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/,
> >>>>> > that would also cut down a lot on the compiler flags (i.e., fewer
> "-I"
> >>>>> > and "-L" flags) :-).
> >>>>> >
> >>>>> > Kapil
> >>>>> >
> >>>>> >>
> >>>>> >> - Jie
> >>>>> >>
> >>>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io>
> wrote:
> >>>>> >>
> >>>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com>
> wrote:
> >>>>> >>>>
> >>>>> >>>>>
> >>>>> >>>>> However, in the current code base, we don't strictly follow the
> >>>>> >>> 3rdparty
> >>>>> >>>>> structure. For example, stout has a dependency on picojson and
> >>>>> >>>>> google-protobuf, but we don't put these two packages inside
> >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>> My understanding is that stout is header only. So it does not
> have to
> >>>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for
> >>>>> bundling
> >>>>> >>>> them if they are used.
> >>>>> >>>
> >>>>> >>>
> >>>>> >>> I don't think being header-only is an excuse to have a broken
> >>>>> >>> installation :-). Further, we don't make it easier for the user
> to get
> >>>>> >>> the 3rdparty binaries either. For example, if the user has a
> different
> >>>>> >>> version of protobuf installed on the system, the compilation of
> any
> >>>>> >>> program that uses stout will fail spectacularly!
> >>>>> >>>
> >>>>> >>> Having said that, the gist here is that we have somewhat
> deviated from
> >>>>> >>> original motivation behind the 3rdparty directory and it would
> be nice
> >>>>> >>> if we can have a flatter structure.
> >>>>> >>>
> >>>>> >>>>
> >>>>> >>>>
> >>>>> >>>> - Jie
> >>>>> >>>>
> >>>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <kapil@mesosphere.io
> >
> >>>>> wrote:
> >>>>> >>>>
> >>>>> >>>>> Hi All,
> >>>>> >>>>>
> >>>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
> >>>>> >>> 3rdparty/.
> >>>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory.
> >>>>> >>>>>
> >>>>> >>>>> I wanted to start some discussion around reorganizing stuff
> inside
> >>>>> >>>>> "3rdparty". I apologize for the length of the email, please
> bear with
> >>>>> >>> me.
> >>>>> >>>>>
> >>>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos
> dependencies
> >>>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
> >>>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies
> (which
> >>>>> may
> >>>>> >>> in
> >>>>> >>>>> turn be Mesos dependencies as well).
> >>>>> >>>>>
> >>>>> >>>>> As I understand, the original motivation was to emphasize that
> >>>>> >>> libprocess
> >>>>> >>>>> is an independent project which depends on "stout", which in
> turn is
> >>>>> >>> also
> >>>>> >>>>> an independent project.
> >>>>> >>>>>
> >>>>> >>>>> However, in the current code base, we don't strictly follow the
> >>>>> >>> 3rdparty
> >>>>> >>>>> structure. For example, stout has a dependency on picojson and
> >>>>> >>>>> google-protobuf, but we don't put these two packages inside
> >>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >>>>> >>>>>
> >>>>> >>>>> In light of these anomalies, I want to propose that we flatten
> out
> >>>>> the
> >>>>> >>>>> 3rdparty directory and put all packages (libprocess, stout,
> protobuf,
> >>>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We
> can
> >>>>> still
> >>>>> >>> use
> >>>>> >>>>> "--with-XYZ=..." to the full extent as needed.
> >>>>> >>>>>
> >>>>> >>>>> To take it a step further, I want to propose that we bring
> libprocess
> >>>>> >>> and
> >>>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e.,
> make
> >>>>> them
> >>>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from
> "third"
> >>>>> >>>>> parties and is used only when "--with-bundled" is defined (by
> >>>>> default).
> >>>>> >>>>> This hierarchy will still allow us to keep libprocess and
> stout as
> >>>>> >>> separate
> >>>>> >>>>> independent projects.
> >>>>> >>>>>
> >>>>> >>>>> The motivation for this proposal came when dealing with
> 3rdparty
> >>>>> >>>>> dependencies for module development. A module developer needs
> access
> >>>>> to
> >>>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact
> >>>>> >>> versions of
> >>>>> >>>>> these packages that Mesos was compiled with.
> >>>>> >>>>>
> >>>>> >>>>> We want to solve this problem by installing module-specific
> 3rdparty
> >>>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make
> install"
> >>>>> (if
> >>>>> >>>>> configured with something like
> >>>>> "--enable-install-module-dependencies").
> >>>>> >>>>> (There is a discussion going on in a separate thread).
> >>>>> >>>>>
> >>>>> >>>>> Further, as of today, when we install Mesos using "make
> install", we
> >>>>> >>>>> install stout headers in "${prefix}/include/stout". However,
> stout
> >>>>> has
> >>>>> >>>>> dependencies on picojson[1] and google-protobuf headers which
> may not
> >>>>> >>> be
> >>>>> >>>>> present on the machine. This leaves stout, and in turn
> libprocess and
> >>>>> >>> Mesos
> >>>>> >>>>> headers, fairly broken. I understand that this issue is
> somewhat
> >>>>> >>> orthogonal
> >>>>> >>>>> to the main issue being discussed in this mail, but I wanted
> to put
> >>>>> it
> >>>>> >>> out
> >>>>> >>>>> since it's related.
> >>>>> >>>>>
> >>>>> >>>>> Any thoughts, comments, concerns are most welcome!
> >>>>> >>>>>
> >>>>> >>>>> Best,
> >>>>> >>>>> Kapil
> >>>>> >>>>>
> >>>>> >>>>>
> >>>>> >>>>> [1]: Picojson issue was resolved as part of
> >>>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h
> into
> >>>>> the
> >>>>> >>>>> include-dir.
> >>>>> >>>>>
> >>>>> >>>
> >>>>>
> >>>>>
> >>
> >>
> >>
> >> --
> >> ~Kevin
> >
> >
> >
> > --
> > Alex
> >
> > Theory is the first term in the Taylor series of practice. -- Thomas M
> > Cover (1992)
>
>
>
> --
> Alex
>
> Theory is the first term in the Taylor series of practice. -- Thomas M
> Cover (1992)
>

Re: Reorganize 3rdparty directory

[Alex Clemmer <cl...@gmail.com>]

Ah, I see now that I coudl have done better by splitting the patch
into two patches -- one where we move everything, and one where we
change the cmake files.

THe important parts of the patch are:

* Adding the contents of `3rdparty/libprocess/3rdparty/CMakeLists.txt`
-> `3rdparty/CMakeLists.txt`; makes since, because we're merging the
two 3rdparty folders, so we only need one CMakeLists.txt. We just dump
the contents unchanged into the other one.
* `3rdparty/libprocess/cmake/Process3rdpartyConfigure.cmake` changing some paths
* `3rdparty/libprocess/CMakeLists.txt` moving a call to `include`.

And that's about it. Everything else is just moving.

On Fri, Feb 26, 2016 at 11:36 AM, Alex Clemmer
<cl...@gmail.com> wrote:
> Folks:
>
> Took about 30 minutes to put together a prototype of the great
> `3rdparty` flattening. Please see the (extremely preliminary)
> review[1] or my working branch[2] for a really close approximation of
> the number of changes to the CMake build system we'd need to support
> this flattening.
>
> Overall, the changes are extremely small. We (apparently) did a pretty
> good job of making the CMake configuration scripts modular and
> consumable by arbitrary projects, so the changes are mostly things
> like "move the config `include` call over there instead of being over
> there" and "change a handful of path variables to reflect the new dir
> structure". (I hope, btw, that it doesn't seem arrogant to say this,
> but I think it is justified given the relatively minor changes in the
> actual CMake code.)
>
> Feel free to remix/take/throw away any subset of this. I don't need
> any credit for it in the final patch if someone marches off in that
> direction. :)
>
> [1] https://reviews.apache.org/r/44099/
> [2] https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd
>
> On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <kl...@gmail.com> wrote:
>> I am also a fan of git submodules in the long term, but avoiding them
>> in the short term.  We should get things organized as we want them
>> first, and then start thinking about pulling libprocess/stout out into
>> submodules later (while also preserving their history!)
>>
>> I disagree with moving libprocess and stout up to the same level as
>> src/. If we want to make sure they don't bleed into Mesos proper, we
>> really should treat them the same as any other 3rdparty code that we
>> depend on.  This will become more relevant when/if we move them into
>> submodules.
>>
>> Given all that, the only real challenge with flattening our 3rdparty
>> dependencies into a single folder should be the changes we have to
>> make to our configure.ac and Makefile.am scripts to know where to look
>> for their dependencies now.  In the end these should be URLs to
>> versioned tarballs that we host somewhere (or git repos that we can
>> have forked and tagged with specific versions).  In the short term
>> these can just be relative paths in the mesos tree though.
>>
>> On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>>> Thanks for bringing it up Alexander!
>>>
>>> I don't have a strong opinion wrt git submodules since I don't have
>>> much experience with them personally. Having said that, I would like
>>> to go conservative on this one (baby steps :-) ).
>>>
>>> Further, I do understand that moving libprocess and stout directories
>>> will be painful for people who already have several branches and will
>>> have conflicts. But I do think, there are some interim solutions as
>>> well (for example, move libprocess/stout to wherever we want, but keep
>>> a symlink from 3rdparty/libprocess, etc, to those new locations for
>>> some time). I am sure there are better solutions out there, but this
>>> should work too.
>>>
>>> Best,
>>> Kapil
>>>
>>> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers
>>> <ew...@groupon.com.invalid> wrote:
>>>> If we go to git submodules, please ensure there are good docs around how to
>>>> update cloned repos.
>>>>
>>>> e.g., From ansible: https://docs.ansible.com/ansible/intro_installation.html
>>>>
>>>> Note when updating ansible, be sure to not only update the source tree, but
>>>> also the “submodules” in git which point at Ansible’s own modules (not the
>>>> same kind of modules, alas).
>>>>
>>>> $ git pull --rebase
>>>> $ git submodule update --init --recursive
>>>>
>>>> Thanks,
>>>>
>>>> - Erik
>>>>
>>>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <al...@mesosphere.io>
>>>> wrote:
>>>>
>>>>> +1
>>>>> I am one who is totally in for that change. It is not only the directories
>>>>> problem, but the structure which has led that the stout tests (which do
>>>>> need to be compiled) are actually managed in the libprocess Makefile, on
>>>>> top of all the things you have already mentioned.
>>>>>
>>>>>
>>>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
>>>>> >
>>>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com> wrote:
>>>>> >> Kapil,
>>>>> >>
>>>>> >> I guess what I want to understand is why the existing structure makes it
>>>>> >> hard for you to do the things that you want to do (installing
>>>>> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as
>>>>> part
>>>>> >> of "make install").
>>>>> >
>>>>> > Let me see if I can answer that :-).
>>>>> >
>>>>> > This is somewhat related. For example, if we want to install protobuf
>>>>> > in 3rdparty/{include,lib} (for module developers to use them without
>>>>> > doing a proper mesos installation), you need to provide the correct
>>>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple
>>>>> > levels of configure.ac, the "--prefix" can at best be generated by
>>>>> > prepending "../../../" to get to the great-grandparent directory. This
>>>>> > is because we have a separate configure.ac which manages
>>>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
>>>>> > but they are not clean.
>>>>> >
>>>>> > Similar thing holds for system-wide installation of these 3rdparty
>>>>> > packages. For example, ideally, we would want to use
>>>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since
>>>>> > they are part of libprocess package, we don't get the correct
>>>>> > directory and have to use either hardwired $pkglibdir, or somehow pass
>>>>> > it from the top-level configure all the way down to
>>>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
>>>>> >
>>>>> >
>>>>> >> The only reason you mentioned in the original email is that "in the
>>>>> current
>>>>> >> code base, we don't strictly follow the 3rdparty structure", which IMO
>>>>> is
>>>>> >> not a very convincing reason for such a big change.
>>>>> >
>>>>> > How about a not so big change? :-). What if we just move
>>>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
>>>>> > stout as is? That is not a big change since we are not touching
>>>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it
>>>>> > will be cleaner and simpler than what we have right now.
>>>>> >
>>>>> > As a later time, we can then consider moving stout out to 3rdparty/
>>>>> > while leaving libprocess as is. But that's something we can decide
>>>>> > later and leave stout as an exception for now.
>>>>> >
>>>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/,
>>>>> > that would also cut down a lot on the compiler flags (i.e., fewer "-I"
>>>>> > and "-L" flags) :-).
>>>>> >
>>>>> > Kapil
>>>>> >
>>>>> >>
>>>>> >> - Jie
>>>>> >>
>>>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>>>>> >>
>>>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
>>>>> >>>>
>>>>> >>>>>
>>>>> >>>>> However, in the current code base, we don't strictly follow the
>>>>> >>> 3rdparty
>>>>> >>>>> structure. For example, stout has a dependency on picojson and
>>>>> >>>>> google-protobuf, but we don't put these two packages inside
>>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> My understanding is that stout is header only. So it does not have to
>>>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for
>>>>> bundling
>>>>> >>>> them if they are used.
>>>>> >>>
>>>>> >>>
>>>>> >>> I don't think being header-only is an excuse to have a broken
>>>>> >>> installation :-). Further, we don't make it easier for the user to get
>>>>> >>> the 3rdparty binaries either. For example, if the user has a different
>>>>> >>> version of protobuf installed on the system, the compilation of any
>>>>> >>> program that uses stout will fail spectacularly!
>>>>> >>>
>>>>> >>> Having said that, the gist here is that we have somewhat deviated from
>>>>> >>> original motivation behind the 3rdparty directory and it would be nice
>>>>> >>> if we can have a flatter structure.
>>>>> >>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> - Jie
>>>>> >>>>
>>>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>>> Hi All,
>>>>> >>>>>
>>>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>>>>> >>> 3rdparty/.
>>>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory.
>>>>> >>>>>
>>>>> >>>>> I wanted to start some discussion around reorganizing stuff inside
>>>>> >>>>> "3rdparty". I apologize for the length of the email, please bear with
>>>>> >>> me.
>>>>> >>>>>
>>>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies
>>>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
>>>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which
>>>>> may
>>>>> >>> in
>>>>> >>>>> turn be Mesos dependencies as well).
>>>>> >>>>>
>>>>> >>>>> As I understand, the original motivation was to emphasize that
>>>>> >>> libprocess
>>>>> >>>>> is an independent project which depends on "stout", which in turn is
>>>>> >>> also
>>>>> >>>>> an independent project.
>>>>> >>>>>
>>>>> >>>>> However, in the current code base, we don't strictly follow the
>>>>> >>> 3rdparty
>>>>> >>>>> structure. For example, stout has a dependency on picojson and
>>>>> >>>>> google-protobuf, but we don't put these two packages inside
>>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>>>> >>>>>
>>>>> >>>>> In light of these anomalies, I want to propose that we flatten out
>>>>> the
>>>>> >>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf,
>>>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can
>>>>> still
>>>>> >>> use
>>>>> >>>>> "--with-XYZ=..." to the full extent as needed.
>>>>> >>>>>
>>>>> >>>>> To take it a step further, I want to propose that we bring libprocess
>>>>> >>> and
>>>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make
>>>>> them
>>>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
>>>>> >>>>> parties and is used only when "--with-bundled" is defined (by
>>>>> default).
>>>>> >>>>> This hierarchy will still allow us to keep libprocess and stout as
>>>>> >>> separate
>>>>> >>>>> independent projects.
>>>>> >>>>>
>>>>> >>>>> The motivation for this proposal came when dealing with 3rdparty
>>>>> >>>>> dependencies for module development. A module developer needs access
>>>>> to
>>>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact
>>>>> >>> versions of
>>>>> >>>>> these packages that Mesos was compiled with.
>>>>> >>>>>
>>>>> >>>>> We want to solve this problem by installing module-specific 3rdparty
>>>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install"
>>>>> (if
>>>>> >>>>> configured with something like
>>>>> "--enable-install-module-dependencies").
>>>>> >>>>> (There is a discussion going on in a separate thread).
>>>>> >>>>>
>>>>> >>>>> Further, as of today, when we install Mesos using "make install", we
>>>>> >>>>> install stout headers in "${prefix}/include/stout". However, stout
>>>>> has
>>>>> >>>>> dependencies on picojson[1] and google-protobuf headers which may not
>>>>> >>> be
>>>>> >>>>> present on the machine. This leaves stout, and in turn libprocess and
>>>>> >>> Mesos
>>>>> >>>>> headers, fairly broken. I understand that this issue is somewhat
>>>>> >>> orthogonal
>>>>> >>>>> to the main issue being discussed in this mail, but I wanted to put
>>>>> it
>>>>> >>> out
>>>>> >>>>> since it's related.
>>>>> >>>>>
>>>>> >>>>> Any thoughts, comments, concerns are most welcome!
>>>>> >>>>>
>>>>> >>>>> Best,
>>>>> >>>>> Kapil
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> [1]: Picojson issue was resolved as part of
>>>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into
>>>>> the
>>>>> >>>>> include-dir.
>>>>> >>>>>
>>>>> >>>
>>>>>
>>>>>
>>
>>
>>
>> --
>> ~Kevin
>
>
>
> --
> Alex
>
> Theory is the first term in the Taylor series of practice. -- Thomas M
> Cover (1992)



-- 
Alex

Theory is the first term in the Taylor series of practice. -- Thomas M
Cover (1992)

Re: Reorganize 3rdparty directory

[Alex Clemmer <cl...@gmail.com>]

Folks:

Took about 30 minutes to put together a prototype of the great
`3rdparty` flattening. Please see the (extremely preliminary)
review[1] or my working branch[2] for a really close approximation of
the number of changes to the CMake build system we'd need to support
this flattening.

Overall, the changes are extremely small. We (apparently) did a pretty
good job of making the CMake configuration scripts modular and
consumable by arbitrary projects, so the changes are mostly things
like "move the config `include` call over there instead of being over
there" and "change a handful of path variables to reflect the new dir
structure". (I hope, btw, that it doesn't seem arrogant to say this,
but I think it is justified given the relatively minor changes in the
actual CMake code.)

Feel free to remix/take/throw away any subset of this. I don't need
any credit for it in the final patch if someone marches off in that
direction. :)

[1] https://reviews.apache.org/r/44099/
[2] https://github.com/hausdorff/mesos/commits/prototype_flattened_3rd

On Thu, Feb 18, 2016 at 12:22 PM, Kevin Klues <kl...@gmail.com> wrote:
> I am also a fan of git submodules in the long term, but avoiding them
> in the short term.  We should get things organized as we want them
> first, and then start thinking about pulling libprocess/stout out into
> submodules later (while also preserving their history!)
>
> I disagree with moving libprocess and stout up to the same level as
> src/. If we want to make sure they don't bleed into Mesos proper, we
> really should treat them the same as any other 3rdparty code that we
> depend on.  This will become more relevant when/if we move them into
> submodules.
>
> Given all that, the only real challenge with flattening our 3rdparty
> dependencies into a single folder should be the changes we have to
> make to our configure.ac and Makefile.am scripts to know where to look
> for their dependencies now.  In the end these should be URLs to
> versioned tarballs that we host somewhere (or git repos that we can
> have forked and tagged with specific versions).  In the short term
> these can just be relative paths in the mesos tree though.
>
> On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>> Thanks for bringing it up Alexander!
>>
>> I don't have a strong opinion wrt git submodules since I don't have
>> much experience with them personally. Having said that, I would like
>> to go conservative on this one (baby steps :-) ).
>>
>> Further, I do understand that moving libprocess and stout directories
>> will be painful for people who already have several branches and will
>> have conflicts. But I do think, there are some interim solutions as
>> well (for example, move libprocess/stout to wherever we want, but keep
>> a symlink from 3rdparty/libprocess, etc, to those new locations for
>> some time). I am sure there are better solutions out there, but this
>> should work too.
>>
>> Best,
>> Kapil
>>
>> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers
>> <ew...@groupon.com.invalid> wrote:
>>> If we go to git submodules, please ensure there are good docs around how to
>>> update cloned repos.
>>>
>>> e.g., From ansible: https://docs.ansible.com/ansible/intro_installation.html
>>>
>>> Note when updating ansible, be sure to not only update the source tree, but
>>> also the “submodules” in git which point at Ansible’s own modules (not the
>>> same kind of modules, alas).
>>>
>>> $ git pull --rebase
>>> $ git submodule update --init --recursive
>>>
>>> Thanks,
>>>
>>> - Erik
>>>
>>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <al...@mesosphere.io>
>>> wrote:
>>>
>>>> +1
>>>> I am one who is totally in for that change. It is not only the directories
>>>> problem, but the structure which has led that the stout tests (which do
>>>> need to be compiled) are actually managed in the libprocess Makefile, on
>>>> top of all the things you have already mentioned.
>>>>
>>>>
>>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
>>>> >
>>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com> wrote:
>>>> >> Kapil,
>>>> >>
>>>> >> I guess what I want to understand is why the existing structure makes it
>>>> >> hard for you to do the things that you want to do (installing
>>>> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as
>>>> part
>>>> >> of "make install").
>>>> >
>>>> > Let me see if I can answer that :-).
>>>> >
>>>> > This is somewhat related. For example, if we want to install protobuf
>>>> > in 3rdparty/{include,lib} (for module developers to use them without
>>>> > doing a proper mesos installation), you need to provide the correct
>>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple
>>>> > levels of configure.ac, the "--prefix" can at best be generated by
>>>> > prepending "../../../" to get to the great-grandparent directory. This
>>>> > is because we have a separate configure.ac which manages
>>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
>>>> > but they are not clean.
>>>> >
>>>> > Similar thing holds for system-wide installation of these 3rdparty
>>>> > packages. For example, ideally, we would want to use
>>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since
>>>> > they are part of libprocess package, we don't get the correct
>>>> > directory and have to use either hardwired $pkglibdir, or somehow pass
>>>> > it from the top-level configure all the way down to
>>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
>>>> >
>>>> >
>>>> >> The only reason you mentioned in the original email is that "in the
>>>> current
>>>> >> code base, we don't strictly follow the 3rdparty structure", which IMO
>>>> is
>>>> >> not a very convincing reason for such a big change.
>>>> >
>>>> > How about a not so big change? :-). What if we just move
>>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
>>>> > stout as is? That is not a big change since we are not touching
>>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it
>>>> > will be cleaner and simpler than what we have right now.
>>>> >
>>>> > As a later time, we can then consider moving stout out to 3rdparty/
>>>> > while leaving libprocess as is. But that's something we can decide
>>>> > later and leave stout as an exception for now.
>>>> >
>>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/,
>>>> > that would also cut down a lot on the compiler flags (i.e., fewer "-I"
>>>> > and "-L" flags) :-).
>>>> >
>>>> > Kapil
>>>> >
>>>> >>
>>>> >> - Jie
>>>> >>
>>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>>>> >>
>>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
>>>> >>>>
>>>> >>>>>
>>>> >>>>> However, in the current code base, we don't strictly follow the
>>>> >>> 3rdparty
>>>> >>>>> structure. For example, stout has a dependency on picojson and
>>>> >>>>> google-protobuf, but we don't put these two packages inside
>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>>> >>>>
>>>> >>>>
>>>> >>>> My understanding is that stout is header only. So it does not have to
>>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for
>>>> bundling
>>>> >>>> them if they are used.
>>>> >>>
>>>> >>>
>>>> >>> I don't think being header-only is an excuse to have a broken
>>>> >>> installation :-). Further, we don't make it easier for the user to get
>>>> >>> the 3rdparty binaries either. For example, if the user has a different
>>>> >>> version of protobuf installed on the system, the compilation of any
>>>> >>> program that uses stout will fail spectacularly!
>>>> >>>
>>>> >>> Having said that, the gist here is that we have somewhat deviated from
>>>> >>> original motivation behind the 3rdparty directory and it would be nice
>>>> >>> if we can have a flatter structure.
>>>> >>>
>>>> >>>>
>>>> >>>>
>>>> >>>> - Jie
>>>> >>>>
>>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io>
>>>> wrote:
>>>> >>>>
>>>> >>>>> Hi All,
>>>> >>>>>
>>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>>>> >>> 3rdparty/.
>>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory.
>>>> >>>>>
>>>> >>>>> I wanted to start some discussion around reorganizing stuff inside
>>>> >>>>> "3rdparty". I apologize for the length of the email, please bear with
>>>> >>> me.
>>>> >>>>>
>>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies
>>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
>>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which
>>>> may
>>>> >>> in
>>>> >>>>> turn be Mesos dependencies as well).
>>>> >>>>>
>>>> >>>>> As I understand, the original motivation was to emphasize that
>>>> >>> libprocess
>>>> >>>>> is an independent project which depends on "stout", which in turn is
>>>> >>> also
>>>> >>>>> an independent project.
>>>> >>>>>
>>>> >>>>> However, in the current code base, we don't strictly follow the
>>>> >>> 3rdparty
>>>> >>>>> structure. For example, stout has a dependency on picojson and
>>>> >>>>> google-protobuf, but we don't put these two packages inside
>>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>>> >>>>>
>>>> >>>>> In light of these anomalies, I want to propose that we flatten out
>>>> the
>>>> >>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf,
>>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can
>>>> still
>>>> >>> use
>>>> >>>>> "--with-XYZ=..." to the full extent as needed.
>>>> >>>>>
>>>> >>>>> To take it a step further, I want to propose that we bring libprocess
>>>> >>> and
>>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make
>>>> them
>>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
>>>> >>>>> parties and is used only when "--with-bundled" is defined (by
>>>> default).
>>>> >>>>> This hierarchy will still allow us to keep libprocess and stout as
>>>> >>> separate
>>>> >>>>> independent projects.
>>>> >>>>>
>>>> >>>>> The motivation for this proposal came when dealing with 3rdparty
>>>> >>>>> dependencies for module development. A module developer needs access
>>>> to
>>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact
>>>> >>> versions of
>>>> >>>>> these packages that Mesos was compiled with.
>>>> >>>>>
>>>> >>>>> We want to solve this problem by installing module-specific 3rdparty
>>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install"
>>>> (if
>>>> >>>>> configured with something like
>>>> "--enable-install-module-dependencies").
>>>> >>>>> (There is a discussion going on in a separate thread).
>>>> >>>>>
>>>> >>>>> Further, as of today, when we install Mesos using "make install", we
>>>> >>>>> install stout headers in "${prefix}/include/stout". However, stout
>>>> has
>>>> >>>>> dependencies on picojson[1] and google-protobuf headers which may not
>>>> >>> be
>>>> >>>>> present on the machine. This leaves stout, and in turn libprocess and
>>>> >>> Mesos
>>>> >>>>> headers, fairly broken. I understand that this issue is somewhat
>>>> >>> orthogonal
>>>> >>>>> to the main issue being discussed in this mail, but I wanted to put
>>>> it
>>>> >>> out
>>>> >>>>> since it's related.
>>>> >>>>>
>>>> >>>>> Any thoughts, comments, concerns are most welcome!
>>>> >>>>>
>>>> >>>>> Best,
>>>> >>>>> Kapil
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> [1]: Picojson issue was resolved as part of
>>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into
>>>> the
>>>> >>>>> include-dir.
>>>> >>>>>
>>>> >>>
>>>>
>>>>
>
>
>
> --
> ~Kevin



-- 
Alex

Theory is the first term in the Taylor series of practice. -- Thomas M
Cover (1992)

Re: Reorganize 3rdparty directory

[Kevin Klues <kl...@gmail.com>]

I am also a fan of git submodules in the long term, but avoiding them
in the short term.  We should get things organized as we want them
first, and then start thinking about pulling libprocess/stout out into
submodules later (while also preserving their history!)

I disagree with moving libprocess and stout up to the same level as
src/. If we want to make sure they don't bleed into Mesos proper, we
really should treat them the same as any other 3rdparty code that we
depend on.  This will become more relevant when/if we move them into
submodules.

Given all that, the only real challenge with flattening our 3rdparty
dependencies into a single folder should be the changes we have to
make to our configure.ac and Makefile.am scripts to know where to look
for their dependencies now.  In the end these should be URLs to
versioned tarballs that we host somewhere (or git repos that we can
have forked and tagged with specific versions).  In the short term
these can just be relative paths in the mesos tree though.

On Tue, Feb 16, 2016 at 1:26 PM, Kapil Arya <ka...@mesosphere.io> wrote:
> Thanks for bringing it up Alexander!
>
> I don't have a strong opinion wrt git submodules since I don't have
> much experience with them personally. Having said that, I would like
> to go conservative on this one (baby steps :-) ).
>
> Further, I do understand that moving libprocess and stout directories
> will be painful for people who already have several branches and will
> have conflicts. But I do think, there are some interim solutions as
> well (for example, move libprocess/stout to wherever we want, but keep
> a symlink from 3rdparty/libprocess, etc, to those new locations for
> some time). I am sure there are better solutions out there, but this
> should work too.
>
> Best,
> Kapil
>
> On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers
> <ew...@groupon.com.invalid> wrote:
>> If we go to git submodules, please ensure there are good docs around how to
>> update cloned repos.
>>
>> e.g., From ansible: https://docs.ansible.com/ansible/intro_installation.html
>>
>> Note when updating ansible, be sure to not only update the source tree, but
>> also the “submodules” in git which point at Ansible’s own modules (not the
>> same kind of modules, alas).
>>
>> $ git pull --rebase
>> $ git submodule update --init --recursive
>>
>> Thanks,
>>
>> - Erik
>>
>> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <al...@mesosphere.io>
>> wrote:
>>
>>> +1
>>> I am one who is totally in for that change. It is not only the directories
>>> problem, but the structure which has led that the stout tests (which do
>>> need to be compiled) are actually managed in the libprocess Makefile, on
>>> top of all the things you have already mentioned.
>>>
>>>
>>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
>>> >
>>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com> wrote:
>>> >> Kapil,
>>> >>
>>> >> I guess what I want to understand is why the existing structure makes it
>>> >> hard for you to do the things that you want to do (installing
>>> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as
>>> part
>>> >> of "make install").
>>> >
>>> > Let me see if I can answer that :-).
>>> >
>>> > This is somewhat related. For example, if we want to install protobuf
>>> > in 3rdparty/{include,lib} (for module developers to use them without
>>> > doing a proper mesos installation), you need to provide the correct
>>> > "--prefix" flag that points to 3rdparty/. However, due to multiple
>>> > levels of configure.ac, the "--prefix" can at best be generated by
>>> > prepending "../../../" to get to the great-grandparent directory. This
>>> > is because we have a separate configure.ac which manages
>>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
>>> > but they are not clean.
>>> >
>>> > Similar thing holds for system-wide installation of these 3rdparty
>>> > packages. For example, ideally, we would want to use
>>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since
>>> > they are part of libprocess package, we don't get the correct
>>> > directory and have to use either hardwired $pkglibdir, or somehow pass
>>> > it from the top-level configure all the way down to
>>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
>>> >
>>> >
>>> >> The only reason you mentioned in the original email is that "in the
>>> current
>>> >> code base, we don't strictly follow the 3rdparty structure", which IMO
>>> is
>>> >> not a very convincing reason for such a big change.
>>> >
>>> > How about a not so big change? :-). What if we just move
>>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
>>> > stout as is? That is not a big change since we are not touching
>>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it
>>> > will be cleaner and simpler than what we have right now.
>>> >
>>> > As a later time, we can then consider moving stout out to 3rdparty/
>>> > while leaving libprocess as is. But that's something we can decide
>>> > later and leave stout as an exception for now.
>>> >
>>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/,
>>> > that would also cut down a lot on the compiler flags (i.e., fewer "-I"
>>> > and "-L" flags) :-).
>>> >
>>> > Kapil
>>> >
>>> >>
>>> >> - Jie
>>> >>
>>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>>> >>
>>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
>>> >>>>
>>> >>>>>
>>> >>>>> However, in the current code base, we don't strictly follow the
>>> >>> 3rdparty
>>> >>>>> structure. For example, stout has a dependency on picojson and
>>> >>>>> google-protobuf, but we don't put these two packages inside
>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>> >>>>
>>> >>>>
>>> >>>> My understanding is that stout is header only. So it does not have to
>>> >>>> bundle 3rdparty libraries. The user of stout is responsible for
>>> bundling
>>> >>>> them if they are used.
>>> >>>
>>> >>>
>>> >>> I don't think being header-only is an excuse to have a broken
>>> >>> installation :-). Further, we don't make it easier for the user to get
>>> >>> the 3rdparty binaries either. For example, if the user has a different
>>> >>> version of protobuf installed on the system, the compilation of any
>>> >>> program that uses stout will fail spectacularly!
>>> >>>
>>> >>> Having said that, the gist here is that we have somewhat deviated from
>>> >>> original motivation behind the 3rdparty directory and it would be nice
>>> >>> if we can have a flatter structure.
>>> >>>
>>> >>>>
>>> >>>>
>>> >>>> - Jie
>>> >>>>
>>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io>
>>> wrote:
>>> >>>>
>>> >>>>> Hi All,
>>> >>>>>
>>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>>> >>> 3rdparty/.
>>> >>>>> (Optionally) Move libprocess/stout to the top-level directory.
>>> >>>>>
>>> >>>>> I wanted to start some discussion around reorganizing stuff inside
>>> >>>>> "3rdparty". I apologize for the length of the email, please bear with
>>> >>> me.
>>> >>>>>
>>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies
>>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
>>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which
>>> may
>>> >>> in
>>> >>>>> turn be Mesos dependencies as well).
>>> >>>>>
>>> >>>>> As I understand, the original motivation was to emphasize that
>>> >>> libprocess
>>> >>>>> is an independent project which depends on "stout", which in turn is
>>> >>> also
>>> >>>>> an independent project.
>>> >>>>>
>>> >>>>> However, in the current code base, we don't strictly follow the
>>> >>> 3rdparty
>>> >>>>> structure. For example, stout has a dependency on picojson and
>>> >>>>> google-protobuf, but we don't put these two packages inside
>>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>> >>>>>
>>> >>>>> In light of these anomalies, I want to propose that we flatten out
>>> the
>>> >>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf,
>>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can
>>> still
>>> >>> use
>>> >>>>> "--with-XYZ=..." to the full extent as needed.
>>> >>>>>
>>> >>>>> To take it a step further, I want to propose that we bring libprocess
>>> >>> and
>>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make
>>> them
>>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
>>> >>>>> parties and is used only when "--with-bundled" is defined (by
>>> default).
>>> >>>>> This hierarchy will still allow us to keep libprocess and stout as
>>> >>> separate
>>> >>>>> independent projects.
>>> >>>>>
>>> >>>>> The motivation for this proposal came when dealing with 3rdparty
>>> >>>>> dependencies for module development. A module developer needs access
>>> to
>>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact
>>> >>> versions of
>>> >>>>> these packages that Mesos was compiled with.
>>> >>>>>
>>> >>>>> We want to solve this problem by installing module-specific 3rdparty
>>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install"
>>> (if
>>> >>>>> configured with something like
>>> "--enable-install-module-dependencies").
>>> >>>>> (There is a discussion going on in a separate thread).
>>> >>>>>
>>> >>>>> Further, as of today, when we install Mesos using "make install", we
>>> >>>>> install stout headers in "${prefix}/include/stout". However, stout
>>> has
>>> >>>>> dependencies on picojson[1] and google-protobuf headers which may not
>>> >>> be
>>> >>>>> present on the machine. This leaves stout, and in turn libprocess and
>>> >>> Mesos
>>> >>>>> headers, fairly broken. I understand that this issue is somewhat
>>> >>> orthogonal
>>> >>>>> to the main issue being discussed in this mail, but I wanted to put
>>> it
>>> >>> out
>>> >>>>> since it's related.
>>> >>>>>
>>> >>>>> Any thoughts, comments, concerns are most welcome!
>>> >>>>>
>>> >>>>> Best,
>>> >>>>> Kapil
>>> >>>>>
>>> >>>>>
>>> >>>>> [1]: Picojson issue was resolved as part of
>>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into
>>> the
>>> >>>>> include-dir.
>>> >>>>>
>>> >>>
>>>
>>>



-- 
~Kevin

Re: Reorganize 3rdparty directory

[Kapil Arya <ka...@mesosphere.io>]

Thanks for bringing it up Alexander!

I don't have a strong opinion wrt git submodules since I don't have
much experience with them personally. Having said that, I would like
to go conservative on this one (baby steps :-) ).

Further, I do understand that moving libprocess and stout directories
will be painful for people who already have several branches and will
have conflicts. But I do think, there are some interim solutions as
well (for example, move libprocess/stout to wherever we want, but keep
a symlink from 3rdparty/libprocess, etc, to those new locations for
some time). I am sure there are better solutions out there, but this
should work too.

Best,
Kapil

On Tue, Feb 16, 2016 at 12:38 PM, Erik Weathers
<ew...@groupon.com.invalid> wrote:
> If we go to git submodules, please ensure there are good docs around how to
> update cloned repos.
>
> e.g., From ansible: https://docs.ansible.com/ansible/intro_installation.html
>
> Note when updating ansible, be sure to not only update the source tree, but
> also the “submodules” in git which point at Ansible’s own modules (not the
> same kind of modules, alas).
>
> $ git pull --rebase
> $ git submodule update --init --recursive
>
> Thanks,
>
> - Erik
>
> On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <al...@mesosphere.io>
> wrote:
>
>> +1
>> I am one who is totally in for that change. It is not only the directories
>> problem, but the structure which has led that the stout tests (which do
>> need to be compiled) are actually managed in the libprocess Makefile, on
>> top of all the things you have already mentioned.
>>
>>
>> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
>> >
>> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com> wrote:
>> >> Kapil,
>> >>
>> >> I guess what I want to understand is why the existing structure makes it
>> >> hard for you to do the things that you want to do (installing
>> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as
>> part
>> >> of "make install").
>> >
>> > Let me see if I can answer that :-).
>> >
>> > This is somewhat related. For example, if we want to install protobuf
>> > in 3rdparty/{include,lib} (for module developers to use them without
>> > doing a proper mesos installation), you need to provide the correct
>> > "--prefix" flag that points to 3rdparty/. However, due to multiple
>> > levels of configure.ac, the "--prefix" can at best be generated by
>> > prepending "../../../" to get to the great-grandparent directory. This
>> > is because we have a separate configure.ac which manages
>> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
>> > but they are not clean.
>> >
>> > Similar thing holds for system-wide installation of these 3rdparty
>> > packages. For example, ideally, we would want to use
>> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since
>> > they are part of libprocess package, we don't get the correct
>> > directory and have to use either hardwired $pkglibdir, or somehow pass
>> > it from the top-level configure all the way down to
>> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
>> >
>> >
>> >> The only reason you mentioned in the original email is that "in the
>> current
>> >> code base, we don't strictly follow the 3rdparty structure", which IMO
>> is
>> >> not a very convincing reason for such a big change.
>> >
>> > How about a not so big change? :-). What if we just move
>> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
>> > stout as is? That is not a big change since we are not touching
>> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it
>> > will be cleaner and simpler than what we have right now.
>> >
>> > As a later time, we can then consider moving stout out to 3rdparty/
>> > while leaving libprocess as is. But that's something we can decide
>> > later and leave stout as an exception for now.
>> >
>> > BTW, if we were to install all the 3rdparty packages in 3rdparty/,
>> > that would also cut down a lot on the compiler flags (i.e., fewer "-I"
>> > and "-L" flags) :-).
>> >
>> > Kapil
>> >
>> >>
>> >> - Jie
>> >>
>> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>> >>
>> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
>> >>>>
>> >>>>>
>> >>>>> However, in the current code base, we don't strictly follow the
>> >>> 3rdparty
>> >>>>> structure. For example, stout has a dependency on picojson and
>> >>>>> google-protobuf, but we don't put these two packages inside
>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> >>>>
>> >>>>
>> >>>> My understanding is that stout is header only. So it does not have to
>> >>>> bundle 3rdparty libraries. The user of stout is responsible for
>> bundling
>> >>>> them if they are used.
>> >>>
>> >>>
>> >>> I don't think being header-only is an excuse to have a broken
>> >>> installation :-). Further, we don't make it easier for the user to get
>> >>> the 3rdparty binaries either. For example, if the user has a different
>> >>> version of protobuf installed on the system, the compilation of any
>> >>> program that uses stout will fail spectacularly!
>> >>>
>> >>> Having said that, the gist here is that we have somewhat deviated from
>> >>> original motivation behind the 3rdparty directory and it would be nice
>> >>> if we can have a flatter structure.
>> >>>
>> >>>>
>> >>>>
>> >>>> - Jie
>> >>>>
>> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io>
>> wrote:
>> >>>>
>> >>>>> Hi All,
>> >>>>>
>> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>> >>> 3rdparty/.
>> >>>>> (Optionally) Move libprocess/stout to the top-level directory.
>> >>>>>
>> >>>>> I wanted to start some discussion around reorganizing stuff inside
>> >>>>> "3rdparty". I apologize for the length of the email, please bear with
>> >>> me.
>> >>>>>
>> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies
>> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
>> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which
>> may
>> >>> in
>> >>>>> turn be Mesos dependencies as well).
>> >>>>>
>> >>>>> As I understand, the original motivation was to emphasize that
>> >>> libprocess
>> >>>>> is an independent project which depends on "stout", which in turn is
>> >>> also
>> >>>>> an independent project.
>> >>>>>
>> >>>>> However, in the current code base, we don't strictly follow the
>> >>> 3rdparty
>> >>>>> structure. For example, stout has a dependency on picojson and
>> >>>>> google-protobuf, but we don't put these two packages inside
>> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> >>>>>
>> >>>>> In light of these anomalies, I want to propose that we flatten out
>> the
>> >>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf,
>> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can
>> still
>> >>> use
>> >>>>> "--with-XYZ=..." to the full extent as needed.
>> >>>>>
>> >>>>> To take it a step further, I want to propose that we bring libprocess
>> >>> and
>> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make
>> them
>> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
>> >>>>> parties and is used only when "--with-bundled" is defined (by
>> default).
>> >>>>> This hierarchy will still allow us to keep libprocess and stout as
>> >>> separate
>> >>>>> independent projects.
>> >>>>>
>> >>>>> The motivation for this proposal came when dealing with 3rdparty
>> >>>>> dependencies for module development. A module developer needs access
>> to
>> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact
>> >>> versions of
>> >>>>> these packages that Mesos was compiled with.
>> >>>>>
>> >>>>> We want to solve this problem by installing module-specific 3rdparty
>> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install"
>> (if
>> >>>>> configured with something like
>> "--enable-install-module-dependencies").
>> >>>>> (There is a discussion going on in a separate thread).
>> >>>>>
>> >>>>> Further, as of today, when we install Mesos using "make install", we
>> >>>>> install stout headers in "${prefix}/include/stout". However, stout
>> has
>> >>>>> dependencies on picojson[1] and google-protobuf headers which may not
>> >>> be
>> >>>>> present on the machine. This leaves stout, and in turn libprocess and
>> >>> Mesos
>> >>>>> headers, fairly broken. I understand that this issue is somewhat
>> >>> orthogonal
>> >>>>> to the main issue being discussed in this mail, but I wanted to put
>> it
>> >>> out
>> >>>>> since it's related.
>> >>>>>
>> >>>>> Any thoughts, comments, concerns are most welcome!
>> >>>>>
>> >>>>> Best,
>> >>>>> Kapil
>> >>>>>
>> >>>>>
>> >>>>> [1]: Picojson issue was resolved as part of
>> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into
>> the
>> >>>>> include-dir.
>> >>>>>
>> >>>
>>
>>

Re: Reorganize 3rdparty directory

[Erik Weathers <ew...@groupon.com.INVALID>]

If we go to git submodules, please ensure there are good docs around how to
update cloned repos.

e.g., From ansible: https://docs.ansible.com/ansible/intro_installation.html

Note when updating ansible, be sure to not only update the source tree, but
also the “submodules” in git which point at Ansible’s own modules (not the
same kind of modules, alas).

$ git pull --rebase
$ git submodule update --init --recursive

Thanks,

- Erik

On Tue, Feb 16, 2016 at 8:54 AM, Alexander Rojas <al...@mesosphere.io>
wrote:

> +1
> I am one who is totally in for that change. It is not only the directories
> problem, but the structure which has led that the stout tests (which do
> need to be compiled) are actually managed in the libprocess Makefile, on
> top of all the things you have already mentioned.
>
>
> > On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
> >
> > On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com> wrote:
> >> Kapil,
> >>
> >> I guess what I want to understand is why the existing structure makes it
> >> hard for you to do the things that you want to do (installing
> >> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as
> part
> >> of "make install").
> >
> > Let me see if I can answer that :-).
> >
> > This is somewhat related. For example, if we want to install protobuf
> > in 3rdparty/{include,lib} (for module developers to use them without
> > doing a proper mesos installation), you need to provide the correct
> > "--prefix" flag that points to 3rdparty/. However, due to multiple
> > levels of configure.ac, the "--prefix" can at best be generated by
> > prepending "../../../" to get to the great-grandparent directory. This
> > is because we have a separate configure.ac which manages
> > 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
> > but they are not clean.
> >
> > Similar thing holds for system-wide installation of these 3rdparty
> > packages. For example, ideally, we would want to use
> > "${pkglibdir}/3rdparty" as a prefix for those packages. However, since
> > they are part of libprocess package, we don't get the correct
> > directory and have to use either hardwired $pkglibdir, or somehow pass
> > it from the top-level configure all the way down to
> > 3rdparty/libprocess/3rdparty/Makefile.am :-(.
> >
> >
> >> The only reason you mentioned in the original email is that "in the
> current
> >> code base, we don't strictly follow the 3rdparty structure", which IMO
> is
> >> not a very convincing reason for such a big change.
> >
> > How about a not so big change? :-). What if we just move
> > 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
> > stout as is? That is not a big change since we are not touching
> > libprocess/stout. Just adjusting Makefiles and I am pretty sure it
> > will be cleaner and simpler than what we have right now.
> >
> > As a later time, we can then consider moving stout out to 3rdparty/
> > while leaving libprocess as is. But that's something we can decide
> > later and leave stout as an exception for now.
> >
> > BTW, if we were to install all the 3rdparty packages in 3rdparty/,
> > that would also cut down a lot on the compiler flags (i.e., fewer "-I"
> > and "-L" flags) :-).
> >
> > Kapil
> >
> >>
> >> - Jie
> >>
> >> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
> >>
> >>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>>
> >>>>> However, in the current code base, we don't strictly follow the
> >>> 3rdparty
> >>>>> structure. For example, stout has a dependency on picojson and
> >>>>> google-protobuf, but we don't put these two packages inside
> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >>>>
> >>>>
> >>>> My understanding is that stout is header only. So it does not have to
> >>>> bundle 3rdparty libraries. The user of stout is responsible for
> bundling
> >>>> them if they are used.
> >>>
> >>>
> >>> I don't think being header-only is an excuse to have a broken
> >>> installation :-). Further, we don't make it easier for the user to get
> >>> the 3rdparty binaries either. For example, if the user has a different
> >>> version of protobuf installed on the system, the compilation of any
> >>> program that uses stout will fail spectacularly!
> >>>
> >>> Having said that, the gist here is that we have somewhat deviated from
> >>> original motivation behind the 3rdparty directory and it would be nice
> >>> if we can have a flatter structure.
> >>>
> >>>>
> >>>>
> >>>> - Jie
> >>>>
> >>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io>
> wrote:
> >>>>
> >>>>> Hi All,
> >>>>>
> >>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
> >>> 3rdparty/.
> >>>>> (Optionally) Move libprocess/stout to the top-level directory.
> >>>>>
> >>>>> I wanted to start some discussion around reorganizing stuff inside
> >>>>> "3rdparty". I apologize for the length of the email, please bear with
> >>> me.
> >>>>>
> >>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies
> >>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
> >>>>> libprocess/3rdparty was to hold all libprocess dependencies (which
> may
> >>> in
> >>>>> turn be Mesos dependencies as well).
> >>>>>
> >>>>> As I understand, the original motivation was to emphasize that
> >>> libprocess
> >>>>> is an independent project which depends on "stout", which in turn is
> >>> also
> >>>>> an independent project.
> >>>>>
> >>>>> However, in the current code base, we don't strictly follow the
> >>> 3rdparty
> >>>>> structure. For example, stout has a dependency on picojson and
> >>>>> google-protobuf, but we don't put these two packages inside
> >>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >>>>>
> >>>>> In light of these anomalies, I want to propose that we flatten out
> the
> >>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf,
> >>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can
> still
> >>> use
> >>>>> "--with-XYZ=..." to the full extent as needed.
> >>>>>
> >>>>> To take it a step further, I want to propose that we bring libprocess
> >>> and
> >>>>> stout out of 3rdparty/ and move them at the top level (i.e., make
> them
> >>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
> >>>>> parties and is used only when "--with-bundled" is defined (by
> default).
> >>>>> This hierarchy will still allow us to keep libprocess and stout as
> >>> separate
> >>>>> independent projects.
> >>>>>
> >>>>> The motivation for this proposal came when dealing with 3rdparty
> >>>>> dependencies for module development. A module developer needs access
> to
> >>>>> protobuf, picojson, glog, etc., and for that matter, the exact
> >>> versions of
> >>>>> these packages that Mesos was compiled with.
> >>>>>
> >>>>> We want to solve this problem by installing module-specific 3rdparty
> >>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install"
> (if
> >>>>> configured with something like
> "--enable-install-module-dependencies").
> >>>>> (There is a discussion going on in a separate thread).
> >>>>>
> >>>>> Further, as of today, when we install Mesos using "make install", we
> >>>>> install stout headers in "${prefix}/include/stout". However, stout
> has
> >>>>> dependencies on picojson[1] and google-protobuf headers which may not
> >>> be
> >>>>> present on the machine. This leaves stout, and in turn libprocess and
> >>> Mesos
> >>>>> headers, fairly broken. I understand that this issue is somewhat
> >>> orthogonal
> >>>>> to the main issue being discussed in this mail, but I wanted to put
> it
> >>> out
> >>>>> since it's related.
> >>>>>
> >>>>> Any thoughts, comments, concerns are most welcome!
> >>>>>
> >>>>> Best,
> >>>>> Kapil
> >>>>>
> >>>>>
> >>>>> [1]: Picojson issue was resolved as part of
> >>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into
> the
> >>>>> include-dir.
> >>>>>
> >>>
>
>

Re: Reorganize 3rdparty directory

[Alexander Rojas <al...@mesosphere.io>]

+1
I am one who is totally in for that change. It is not only the directories problem, but the structure which has led that the stout tests (which do need to be compiled) are actually managed in the libprocess Makefile, on top of all the things you have already mentioned.


> On 09 Feb 2016, at 17:53, Kapil Arya <ka...@mesosphere.io> wrote:
> 
> On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com> wrote:
>> Kapil,
>> 
>> I guess what I want to understand is why the existing structure makes it
>> hard for you to do the things that you want to do (installing
>> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as part
>> of "make install").
> 
> Let me see if I can answer that :-).
> 
> This is somewhat related. For example, if we want to install protobuf
> in 3rdparty/{include,lib} (for module developers to use them without
> doing a proper mesos installation), you need to provide the correct
> "--prefix" flag that points to 3rdparty/. However, due to multiple
> levels of configure.ac, the "--prefix" can at best be generated by
> prepending "../../../" to get to the great-grandparent directory. This
> is because we have a separate configure.ac which manages
> 3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
> but they are not clean.
> 
> Similar thing holds for system-wide installation of these 3rdparty
> packages. For example, ideally, we would want to use
> "${pkglibdir}/3rdparty" as a prefix for those packages. However, since
> they are part of libprocess package, we don't get the correct
> directory and have to use either hardwired $pkglibdir, or somehow pass
> it from the top-level configure all the way down to
> 3rdparty/libprocess/3rdparty/Makefile.am :-(.
> 
> 
>> The only reason you mentioned in the original email is that "in the current
>> code base, we don't strictly follow the 3rdparty structure", which IMO is
>> not a very convincing reason for such a big change.
> 
> How about a not so big change? :-). What if we just move
> 3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
> stout as is? That is not a big change since we are not touching
> libprocess/stout. Just adjusting Makefiles and I am pretty sure it
> will be cleaner and simpler than what we have right now.
> 
> As a later time, we can then consider moving stout out to 3rdparty/
> while leaving libprocess as is. But that's something we can decide
> later and leave stout as an exception for now.
> 
> BTW, if we were to install all the 3rdparty packages in 3rdparty/,
> that would also cut down a lot on the compiler flags (i.e., fewer "-I"
> and "-L" flags) :-).
> 
> Kapil
> 
>> 
>> - Jie
>> 
>> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>> 
>>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
>>>> 
>>>>> 
>>>>> However, in the current code base, we don't strictly follow the
>>> 3rdparty
>>>>> structure. For example, stout has a dependency on picojson and
>>>>> google-protobuf, but we don't put these two packages inside
>>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>>> 
>>>> 
>>>> My understanding is that stout is header only. So it does not have to
>>>> bundle 3rdparty libraries. The user of stout is responsible for bundling
>>>> them if they are used.
>>> 
>>> 
>>> I don't think being header-only is an excuse to have a broken
>>> installation :-). Further, we don't make it easier for the user to get
>>> the 3rdparty binaries either. For example, if the user has a different
>>> version of protobuf installed on the system, the compilation of any
>>> program that uses stout will fail spectacularly!
>>> 
>>> Having said that, the gist here is that we have somewhat deviated from
>>> original motivation behind the 3rdparty directory and it would be nice
>>> if we can have a flatter structure.
>>> 
>>>> 
>>>> 
>>>> - Jie
>>>> 
>>>> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>>>> 
>>>>> Hi All,
>>>>> 
>>>>> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>>> 3rdparty/.
>>>>> (Optionally) Move libprocess/stout to the top-level directory.
>>>>> 
>>>>> I wanted to start some discussion around reorganizing stuff inside
>>>>> "3rdparty". I apologize for the length of the email, please bear with
>>> me.
>>>>> 
>>>>> Traditionally, 3rdparty has been used to hold all Mesos dependencies
>>>>> (zookeeper, libprocess, protobuf, stout, etc.). Further,
>>>>> libprocess/3rdparty was to hold all libprocess dependencies (which may
>>> in
>>>>> turn be Mesos dependencies as well).
>>>>> 
>>>>> As I understand, the original motivation was to emphasize that
>>> libprocess
>>>>> is an independent project which depends on "stout", which in turn is
>>> also
>>>>> an independent project.
>>>>> 
>>>>> However, in the current code base, we don't strictly follow the
>>> 3rdparty
>>>>> structure. For example, stout has a dependency on picojson and
>>>>> google-protobuf, but we don't put these two packages inside
>>>>> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>>>>> 
>>>>> In light of these anomalies, I want to propose that we flatten out the
>>>>> 3rdparty directory and put all packages (libprocess, stout, protobuf,
>>>>> picojson, zookeeper, etc.) at the same level in 3rdparty. We can still
>>> use
>>>>> "--with-XYZ=..." to the full extent as needed.
>>>>> 
>>>>> To take it a step further, I want to propose that we bring libprocess
>>> and
>>>>> stout out of 3rdparty/ and move them at the top level (i.e., make them
>>>>> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
>>>>> parties and is used only when "--with-bundled" is defined (by default).
>>>>> This hierarchy will still allow us to keep libprocess and stout as
>>> separate
>>>>> independent projects.
>>>>> 
>>>>> The motivation for this proposal came when dealing with 3rdparty
>>>>> dependencies for module development. A module developer needs access to
>>>>> protobuf, picojson, glog, etc., and for that matter, the exact
>>> versions of
>>>>> these packages that Mesos was compiled with.
>>>>> 
>>>>> We want to solve this problem by installing module-specific 3rdparty
>>>>> dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
>>>>> configured with something like "--enable-install-module-dependencies").
>>>>> (There is a discussion going on in a separate thread).
>>>>> 
>>>>> Further, as of today, when we install Mesos using "make install", we
>>>>> install stout headers in "${prefix}/include/stout". However, stout has
>>>>> dependencies on picojson[1] and google-protobuf headers which may not
>>> be
>>>>> present on the machine. This leaves stout, and in turn libprocess and
>>> Mesos
>>>>> headers, fairly broken. I understand that this issue is somewhat
>>> orthogonal
>>>>> to the main issue being discussed in this mail, but I wanted to put it
>>> out
>>>>> since it's related.
>>>>> 
>>>>> Any thoughts, comments, concerns are most welcome!
>>>>> 
>>>>> Best,
>>>>> Kapil
>>>>> 
>>>>> 
>>>>> [1]: Picojson issue was resolved as part of
>>>>> https://reviews.apache.org/r/41424/ which installs picojson.h into the
>>>>> include-dir.
>>>>> 
>>> 

Re: Reorganize 3rdparty directory

[Kapil Arya <ka...@mesosphere.io>]

On Tue, Feb 9, 2016 at 8:23 PM, Jie Yu <yu...@gmail.com> wrote:
> Kapil,
>
> I guess what I want to understand is why the existing structure makes it
> hard for you to do the things that you want to do (installing
> module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as part
> of "make install").

Let me see if I can answer that :-).

This is somewhat related. For example, if we want to install protobuf
in 3rdparty/{include,lib} (for module developers to use them without
doing a proper mesos installation), you need to provide the correct
"--prefix" flag that points to 3rdparty/. However, due to multiple
levels of configure.ac, the "--prefix" can at best be generated by
prepending "../../../" to get to the great-grandparent directory. This
is because we have a separate configure.ac which manages
3rdparty/libprocess/3rdparty/Makefile.am. There are ways around it,
but they are not clean.

Similar thing holds for system-wide installation of these 3rdparty
packages. For example, ideally, we would want to use
"${pkglibdir}/3rdparty" as a prefix for those packages. However, since
they are part of libprocess package, we don't get the correct
directory and have to use either hardwired $pkglibdir, or somehow pass
it from the top-level configure all the way down to
3rdparty/libprocess/3rdparty/Makefile.am :-(.


> The only reason you mentioned in the original email is that "in the current
> code base, we don't strictly follow the 3rdparty structure", which IMO is
> not a very convincing reason for such a big change.

How about a not so big change? :-). What if we just move
3rdparty/libprocess/3rdparty/* stuff out to 3rdparty/ while leaving
stout as is? That is not a big change since we are not touching
libprocess/stout. Just adjusting Makefiles and I am pretty sure it
will be cleaner and simpler than what we have right now.

As a later time, we can then consider moving stout out to 3rdparty/
while leaving libprocess as is. But that's something we can decide
later and leave stout as an exception for now.

BTW, if we were to install all the 3rdparty packages in 3rdparty/,
that would also cut down a lot on the compiler flags (i.e., fewer "-I"
and "-L" flags) :-).

Kapil

>
> - Jie
>
> On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>
>> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
>> >
>> > >
>> > > However, in the current code base, we don't strictly follow the
>> 3rdparty
>> > > structure. For example, stout has a dependency on picojson and
>> > > google-protobuf, but we don't put these two packages inside
>> > > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> >
>> >
>> > My understanding is that stout is header only. So it does not have to
>> > bundle 3rdparty libraries. The user of stout is responsible for bundling
>> > them if they are used.
>>
>>
>> I don't think being header-only is an excuse to have a broken
>> installation :-). Further, we don't make it easier for the user to get
>> the 3rdparty binaries either. For example, if the user has a different
>> version of protobuf installed on the system, the compilation of any
>> program that uses stout will fail spectacularly!
>>
>> Having said that, the gist here is that we have somewhat deviated from
>> original motivation behind the 3rdparty directory and it would be nice
>> if we can have a flatter structure.
>>
>> >
>> >
>> > - Jie
>> >
>> > On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>> >
>> > > Hi All,
>> > >
>> > > TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
>> 3rdparty/.
>> > > (Optionally) Move libprocess/stout to the top-level directory.
>> > >
>> > > I wanted to start some discussion around reorganizing stuff inside
>> > > "3rdparty". I apologize for the length of the email, please bear with
>> me.
>> > >
>> > > Traditionally, 3rdparty has been used to hold all Mesos dependencies
>> > > (zookeeper, libprocess, protobuf, stout, etc.). Further,
>> > > libprocess/3rdparty was to hold all libprocess dependencies (which may
>> in
>> > > turn be Mesos dependencies as well).
>> > >
>> > > As I understand, the original motivation was to emphasize that
>> libprocess
>> > > is an independent project which depends on "stout", which in turn is
>> also
>> > > an independent project.
>> > >
>> > > However, in the current code base, we don't strictly follow the
>> 3rdparty
>> > > structure. For example, stout has a dependency on picojson and
>> > > google-protobuf, but we don't put these two packages inside
>> > > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>> > >
>> > > In light of these anomalies, I want to propose that we flatten out the
>> > > 3rdparty directory and put all packages (libprocess, stout, protobuf,
>> > > picojson, zookeeper, etc.) at the same level in 3rdparty. We can still
>> use
>> > > "--with-XYZ=..." to the full extent as needed.
>> > >
>> > > To take it a step further, I want to propose that we bring libprocess
>> and
>> > > stout out of 3rdparty/ and move them at the top level (i.e., make them
>> > > peers of src/). That way, all code in 3rdparty/ is stuff from "third"
>> > > parties and is used only when "--with-bundled" is defined (by default).
>> > > This hierarchy will still allow us to keep libprocess and stout as
>> separate
>> > > independent projects.
>> > >
>> > > The motivation for this proposal came when dealing with 3rdparty
>> > > dependencies for module development. A module developer needs access to
>> > > protobuf, picojson, glog, etc., and for that matter, the exact
>> versions of
>> > > these packages that Mesos was compiled with.
>> > >
>> > > We want to solve this problem by installing module-specific 3rdparty
>> > > dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
>> > > configured with something like "--enable-install-module-dependencies").
>> > > (There is a discussion going on in a separate thread).
>> > >
>> > > Further, as of today, when we install Mesos using "make install", we
>> > > install stout headers in "${prefix}/include/stout". However, stout has
>> > > dependencies on picojson[1] and google-protobuf headers which may not
>> be
>> > > present on the machine. This leaves stout, and in turn libprocess and
>> Mesos
>> > > headers, fairly broken. I understand that this issue is somewhat
>> orthogonal
>> > > to the main issue being discussed in this mail, but I wanted to put it
>> out
>> > > since it's related.
>> > >
>> > > Any thoughts, comments, concerns are most welcome!
>> > >
>> > > Best,
>> > > Kapil
>> > >
>> > >
>> > > [1]: Picojson issue was resolved as part of
>> > > https://reviews.apache.org/r/41424/ which installs picojson.h into the
>> > > include-dir.
>> > >
>>

Re: Reorganize 3rdparty directory

[Jie Yu <yu...@gmail.com>]

Kapil,

I guess what I want to understand is why the existing structure makes it
hard for you to do the things that you want to do (installing
module-specific 3rdparty dependencies into "${pkglibdir}/3rdparty" as part
of "make install").

The only reason you mentioned in the original email is that "in the current
code base, we don't strictly follow the 3rdparty structure", which IMO is
not a very convincing reason for such a big change.

- Jie

On Tue, Feb 9, 2016 at 5:04 PM, Kapil Arya <ka...@mesosphere.io> wrote:

> On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
> >
> > >
> > > However, in the current code base, we don't strictly follow the
> 3rdparty
> > > structure. For example, stout has a dependency on picojson and
> > > google-protobuf, but we don't put these two packages inside
> > > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >
> >
> > My understanding is that stout is header only. So it does not have to
> > bundle 3rdparty libraries. The user of stout is responsible for bundling
> > them if they are used.
>
>
> I don't think being header-only is an excuse to have a broken
> installation :-). Further, we don't make it easier for the user to get
> the 3rdparty binaries either. For example, if the user has a different
> version of protobuf installed on the system, the compilation of any
> program that uses stout will fail spectacularly!
>
> Having said that, the gist here is that we have somewhat deviated from
> original motivation behind the 3rdparty directory and it would be nice
> if we can have a flatter structure.
>
> >
> >
> > - Jie
> >
> > On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> wrote:
> >
> > > Hi All,
> > >
> > > TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into
> 3rdparty/.
> > > (Optionally) Move libprocess/stout to the top-level directory.
> > >
> > > I wanted to start some discussion around reorganizing stuff inside
> > > "3rdparty". I apologize for the length of the email, please bear with
> me.
> > >
> > > Traditionally, 3rdparty has been used to hold all Mesos dependencies
> > > (zookeeper, libprocess, protobuf, stout, etc.). Further,
> > > libprocess/3rdparty was to hold all libprocess dependencies (which may
> in
> > > turn be Mesos dependencies as well).
> > >
> > > As I understand, the original motivation was to emphasize that
> libprocess
> > > is an independent project which depends on "stout", which in turn is
> also
> > > an independent project.
> > >
> > > However, in the current code base, we don't strictly follow the
> 3rdparty
> > > structure. For example, stout has a dependency on picojson and
> > > google-protobuf, but we don't put these two packages inside
> > > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> > >
> > > In light of these anomalies, I want to propose that we flatten out the
> > > 3rdparty directory and put all packages (libprocess, stout, protobuf,
> > > picojson, zookeeper, etc.) at the same level in 3rdparty. We can still
> use
> > > "--with-XYZ=..." to the full extent as needed.
> > >
> > > To take it a step further, I want to propose that we bring libprocess
> and
> > > stout out of 3rdparty/ and move them at the top level (i.e., make them
> > > peers of src/). That way, all code in 3rdparty/ is stuff from "third"
> > > parties and is used only when "--with-bundled" is defined (by default).
> > > This hierarchy will still allow us to keep libprocess and stout as
> separate
> > > independent projects.
> > >
> > > The motivation for this proposal came when dealing with 3rdparty
> > > dependencies for module development. A module developer needs access to
> > > protobuf, picojson, glog, etc., and for that matter, the exact
> versions of
> > > these packages that Mesos was compiled with.
> > >
> > > We want to solve this problem by installing module-specific 3rdparty
> > > dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
> > > configured with something like "--enable-install-module-dependencies").
> > > (There is a discussion going on in a separate thread).
> > >
> > > Further, as of today, when we install Mesos using "make install", we
> > > install stout headers in "${prefix}/include/stout". However, stout has
> > > dependencies on picojson[1] and google-protobuf headers which may not
> be
> > > present on the machine. This leaves stout, and in turn libprocess and
> Mesos
> > > headers, fairly broken. I understand that this issue is somewhat
> orthogonal
> > > to the main issue being discussed in this mail, but I wanted to put it
> out
> > > since it's related.
> > >
> > > Any thoughts, comments, concerns are most welcome!
> > >
> > > Best,
> > > Kapil
> > >
> > >
> > > [1]: Picojson issue was resolved as part of
> > > https://reviews.apache.org/r/41424/ which installs picojson.h into the
> > > include-dir.
> > >
>

Re: Reorganize 3rdparty directory

[Kapil Arya <ka...@mesosphere.io>]

On Tue, Feb 9, 2016 at 7:20 PM, Jie Yu <yu...@gmail.com> wrote:
>
> >
> > However, in the current code base, we don't strictly follow the 3rdparty
> > structure. For example, stout has a dependency on picojson and
> > google-protobuf, but we don't put these two packages inside
> > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>
>
> My understanding is that stout is header only. So it does not have to
> bundle 3rdparty libraries. The user of stout is responsible for bundling
> them if they are used.


I don't think being header-only is an excuse to have a broken
installation :-). Further, we don't make it easier for the user to get
the 3rdparty binaries either. For example, if the user has a different
version of protobuf installed on the system, the compilation of any
program that uses stout will fail spectacularly!

Having said that, the gist here is that we have somewhat deviated from
original motivation behind the 3rdparty directory and it would be nice
if we can have a flatter structure.

>
>
> - Jie
>
> On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> wrote:
>
> > Hi All,
> >
> > TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into 3rdparty/.
> > (Optionally) Move libprocess/stout to the top-level directory.
> >
> > I wanted to start some discussion around reorganizing stuff inside
> > "3rdparty". I apologize for the length of the email, please bear with me.
> >
> > Traditionally, 3rdparty has been used to hold all Mesos dependencies
> > (zookeeper, libprocess, protobuf, stout, etc.). Further,
> > libprocess/3rdparty was to hold all libprocess dependencies (which may in
> > turn be Mesos dependencies as well).
> >
> > As I understand, the original motivation was to emphasize that libprocess
> > is an independent project which depends on "stout", which in turn is also
> > an independent project.
> >
> > However, in the current code base, we don't strictly follow the 3rdparty
> > structure. For example, stout has a dependency on picojson and
> > google-protobuf, but we don't put these two packages inside
> > 3rdparty/libprocess/3rdparty/stout/3rdparty/.
> >
> > In light of these anomalies, I want to propose that we flatten out the
> > 3rdparty directory and put all packages (libprocess, stout, protobuf,
> > picojson, zookeeper, etc.) at the same level in 3rdparty. We can still use
> > "--with-XYZ=..." to the full extent as needed.
> >
> > To take it a step further, I want to propose that we bring libprocess and
> > stout out of 3rdparty/ and move them at the top level (i.e., make them
> > peers of src/). That way, all code in 3rdparty/ is stuff from "third"
> > parties and is used only when "--with-bundled" is defined (by default).
> > This hierarchy will still allow us to keep libprocess and stout as separate
> > independent projects.
> >
> > The motivation for this proposal came when dealing with 3rdparty
> > dependencies for module development. A module developer needs access to
> > protobuf, picojson, glog, etc., and for that matter, the exact versions of
> > these packages that Mesos was compiled with.
> >
> > We want to solve this problem by installing module-specific 3rdparty
> > dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
> > configured with something like "--enable-install-module-dependencies").
> > (There is a discussion going on in a separate thread).
> >
> > Further, as of today, when we install Mesos using "make install", we
> > install stout headers in "${prefix}/include/stout". However, stout has
> > dependencies on picojson[1] and google-protobuf headers which may not be
> > present on the machine. This leaves stout, and in turn libprocess and Mesos
> > headers, fairly broken. I understand that this issue is somewhat orthogonal
> > to the main issue being discussed in this mail, but I wanted to put it out
> > since it's related.
> >
> > Any thoughts, comments, concerns are most welcome!
> >
> > Best,
> > Kapil
> >
> >
> > [1]: Picojson issue was resolved as part of
> > https://reviews.apache.org/r/41424/ which installs picojson.h into the
> > include-dir.
> >

Re: Reorganize 3rdparty directory

[Jie Yu <yu...@gmail.com>]

>
> However, in the current code base, we don't strictly follow the 3rdparty
> structure. For example, stout has a dependency on picojson and
> google-protobuf, but we don't put these two packages inside
> 3rdparty/libprocess/3rdparty/stout/3rdparty/.


My understanding is that stout is header only. So it does not have to
bundle 3rdparty libraries. The user of stout is responsible for bundling
them if they are used.

- Jie

On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> wrote:

> Hi All,
>
> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into 3rdparty/.
> (Optionally) Move libprocess/stout to the top-level directory.
>
> I wanted to start some discussion around reorganizing stuff inside
> "3rdparty". I apologize for the length of the email, please bear with me.
>
> Traditionally, 3rdparty has been used to hold all Mesos dependencies
> (zookeeper, libprocess, protobuf, stout, etc.). Further,
> libprocess/3rdparty was to hold all libprocess dependencies (which may in
> turn be Mesos dependencies as well).
>
> As I understand, the original motivation was to emphasize that libprocess
> is an independent project which depends on "stout", which in turn is also
> an independent project.
>
> However, in the current code base, we don't strictly follow the 3rdparty
> structure. For example, stout has a dependency on picojson and
> google-protobuf, but we don't put these two packages inside
> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>
> In light of these anomalies, I want to propose that we flatten out the
> 3rdparty directory and put all packages (libprocess, stout, protobuf,
> picojson, zookeeper, etc.) at the same level in 3rdparty. We can still use
> "--with-XYZ=..." to the full extent as needed.
>
> To take it a step further, I want to propose that we bring libprocess and
> stout out of 3rdparty/ and move them at the top level (i.e., make them
> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
> parties and is used only when "--with-bundled" is defined (by default).
> This hierarchy will still allow us to keep libprocess and stout as separate
> independent projects.
>
> The motivation for this proposal came when dealing with 3rdparty
> dependencies for module development. A module developer needs access to
> protobuf, picojson, glog, etc., and for that matter, the exact versions of
> these packages that Mesos was compiled with.
>
> We want to solve this problem by installing module-specific 3rdparty
> dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
> configured with something like "--enable-install-module-dependencies").
> (There is a discussion going on in a separate thread).
>
> Further, as of today, when we install Mesos using "make install", we
> install stout headers in "${prefix}/include/stout". However, stout has
> dependencies on picojson[1] and google-protobuf headers which may not be
> present on the machine. This leaves stout, and in turn libprocess and Mesos
> headers, fairly broken. I understand that this issue is somewhat orthogonal
> to the main issue being discussed in this mail, but I wanted to put it out
> since it's related.
>
> Any thoughts, comments, concerns are most welcome!
>
> Best,
> Kapil
>
>
> [1]: Picojson issue was resolved as part of
> https://reviews.apache.org/r/41424/ which installs picojson.h into the
> include-dir.
>

Re: Reorganize 3rdparty directory

[Marco Massenzio <m....@gmail.com>]

This is a great initiative and something I wanted to do since the first day
I did `git clone` for Mesos :)
Happy to help wherever I can, Kapil - just say the word!

In my opinion, we should take full advantage of git submodules and move
stout/libprocess out into their own repositories (which, incidentally,
already exist [0, 1]) and move all the binaries (*.tar.gz) out of our repo:
I never quite understood why we're carrying along zookeeper and protobuf
(just to mention two), which I'm sure both Apache and Google can take good
care of (and we just install as part of the build process - or whatever).

I've recently been using git submodules and they're pretty awesome, also
allowing other teams who may want to use either library the opportunity to
do so (and contribute back - without having to be part of Mesos proper).

So, totally +1 to this idea.

[0] https://github.com/3rdparty/stout
[1] https://github.com/3rdparty/libprocess

-- 
*Marco Massenzio*
http://codetrips.com

On Tue, Feb 9, 2016 at 4:14 PM, Kapil Arya <ka...@mesosphere.io> wrote:

> Hi All,
>
> TLDR: Move everything from 3rdparty/libprocess/3rdparty/* into 3rdparty/.
> (Optionally) Move libprocess/stout to the top-level directory.
>
> I wanted to start some discussion around reorganizing stuff inside
> "3rdparty". I apologize for the length of the email, please bear with me.
>
> Traditionally, 3rdparty has been used to hold all Mesos dependencies
> (zookeeper, libprocess, protobuf, stout, etc.). Further,
> libprocess/3rdparty was to hold all libprocess dependencies (which may in
> turn be Mesos dependencies as well).
>
> As I understand, the original motivation was to emphasize that libprocess
> is an independent project which depends on "stout", which in turn is also
> an independent project.
>
> However, in the current code base, we don't strictly follow the 3rdparty
> structure. For example, stout has a dependency on picojson and
> google-protobuf, but we don't put these two packages inside
> 3rdparty/libprocess/3rdparty/stout/3rdparty/.
>
> In light of these anomalies, I want to propose that we flatten out the
> 3rdparty directory and put all packages (libprocess, stout, protobuf,
> picojson, zookeeper, etc.) at the same level in 3rdparty. We can still use
> "--with-XYZ=..." to the full extent as needed.
>
> To take it a step further, I want to propose that we bring libprocess and
> stout out of 3rdparty/ and move them at the top level (i.e., make them
> peers of src/). That way, all code in 3rdparty/ is stuff from "third"
> parties and is used only when "--with-bundled" is defined (by default).
> This hierarchy will still allow us to keep libprocess and stout as separate
> independent projects.
>
> The motivation for this proposal came when dealing with 3rdparty
> dependencies for module development. A module developer needs access to
> protobuf, picojson, glog, etc., and for that matter, the exact versions of
> these packages that Mesos was compiled with.
>
> We want to solve this problem by installing module-specific 3rdparty
> dependencies into "${pkglibdir}/3rdparty" as part of "make install" (if
> configured with something like "--enable-install-module-dependencies").
> (There is a discussion going on in a separate thread).
>
> Further, as of today, when we install Mesos using "make install", we
> install stout headers in "${prefix}/include/stout". However, stout has
> dependencies on picojson[1] and google-protobuf headers which may not be
> present on the machine. This leaves stout, and in turn libprocess and Mesos
> headers, fairly broken. I understand that this issue is somewhat orthogonal
> to the main issue being discussed in this mail, but I wanted to put it out
> since it's related.
>
> Any thoughts, comments, concerns are most welcome!
>
> Best,
> Kapil
>
>
> [1]: Picojson issue was resolved as part of
> https://reviews.apache.org/r/41424/ which installs picojson.h into the
> include-dir.
>