Re: Separate Compilation of Tests

[Joris Van Remoortere <jo...@mesosphere.io>]

@mpark
Can you clarify if your goal is to:
1) refactor the source code (maybe including things like cluster.hpp) so
that general compilation of any file is faster. Examples include allowing
non-header-only stout, splitting out some files, moving some more stuff to
cpps.
or 2) splitting out the files in the cmake / autoconf so that it's possible
to compile a single test file?

Although 2) is probably the easiest way to improved performance I was under
the impression there were some arguments ~2 years ago around wanting to
have all the tests in a single binary.

I think any effort spent 1) will be valuable regardless. We've talked about
a bunch of tasks in #1 and just needed time / commitment to them.

Some people have also suggested improved tooling. For example the gold
linker.
You should also see if you are ever IO bound. Debug builds generate
binaries so large that you actually can become IO bound. There are ways to
get around that too. There have been times where an optimized build has
been faster than a debug build.

Joris

—
*Joris Van Remoortere*
Mesosphere

On Tue, Sep 27, 2016 at 6:25 PM, Michael Park <mp...@apache.org> wrote:

> On Tue, Sep 27, 2016 at 12:21 PM, Alexander Rojas <alexander@mesosphere.io
> >
> wrote:
>
> > The workaround I use is to go to the folder of the makefile root
> > directory, so for
> > Mesos, and then use the specific target, for example:
> >
> > For mesos-tests
> > cd ${MESOS_SRC_ROOT}/build/src
> > make -j4 mesos-tests && ./mesos-tests
> >
> > for libprocess
> > cd ${MESOS_SRC_ROOT}/build/3rdparty/libprocess
> > make -j4 libprocess-tests && ./libprocess-tests
> >
> > for stout
> > cd ${MESOS_SRC_ROOT}/build/3rdparty/stout
> > make -j4 stout-tests && ./stout-tests
> >
>
> Alexander, I'm aware of these targets, and I guess my claim is that these
> are not fine-grained enough.
> Yes, I can work on a stout change and only run stout tests to iterate
> without compiling/running all of the mesos tests.
>
> But I'm talking about times when I'm making mesos changes (let's say
> involving a stout change),
> I don't want to have to recompile basically all of the tests per iteration.
>
>
> > I don’t think it would be hard to add top level targets that just do that
> > from the top level.
> >
>
> I imagine it's probably not either, but it's not what I'm looking for in
> this thread.
>
> I however would like to have a target that compiles but not runs the tests.
> >
>
> What do you mean by this? Are you looking for `make tests`?
> As far as I'm aware, you can do `make tests`, and `make check` at every
> level,
> as opposed to the commands you're running to achieve what seems like the
> same thing to me.
>
>
> > > On 25 Sep 2016, at 15:56, Michael Park <mp...@apache.org> wrote:
> > >
> > > Hello,
> > >
> > > I would like to propose a change in our build to help us improve
> > developer
> > > efficiency.
> > > The proposal is to support separate compilation of unit tests.
> > >
> > > Currently, we have the old approach of invoking `make check -j N
> > > GTEST_FILTER=""`, or a newer option of doing `make tests -j N`. From
> what
> > > I've heard the latter is slightly faster. However, when someone is
> > > developing a specific feature, it's likely that they would like to make
> > > changes and iterate on a single test file. In this workflow, having to
> > > compile (virtually) __all__ of the tests is very burdensome. This
> > situation
> > > is not so bad if you're working in a very isolated part of the
> codebase,
> > > but it gets to be pretty bad if you're experimenting with parts that
> are
> > > widely used.
> > >
> > > An example of a workflow I'm aiming for would look something like:
> > >
> > >   1. write some code...
> > >   2. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   3. fix compilation errors...
> > >   4. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   5. change some stuff...
> > >   6. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   7. debug...
> > >   8. `make check master_tests`  // compile and test
> > >   `src/tests/master_tests.cpp`
> > >   9. alright, looks good. `make check`
> > >
> > > I have 0 attachment to the `make check master_tests` syntax. It'll be a
> > > different syntax for CMake anyways. I just think that the ability to
> > > perform separate compilation of tests will be immensely useful.
> > >
> > > Some numbers to justify what I'm proposing:
> > >
> > >   - `make -j 8` on my laptop takes roughly 10 mins.
> > >   - `make tests -j 8` takes about 30 mins.
> > >
> > > Of course, not every change you make triggers all of the tests to
> > > recompile. But if you change components that are widely used, it does
> end
> > > up recompiling virtually everything. Under these circumstances, I would
> > > love for each iteration to be 11 mins (10 mins + __at most__ 1 min to
> > > compile the single test), rather than 30 mins.
> > >
> > > NOTE: My laptop is expensive... some people even use machines with 64
> > cores
> > > or whatever to compile Mesos. Not everyone has access to this kind of
> > > equipment. We should strive for something better than "throw more money
> > at
> > > it".
> > >
> > > The goal of this thread for me is the following:
> > >  (1) Capture whether this is something most people experience, or
> whether
> > > I'm just doing it wrong
> > >  (2) If most people do experience this inefficiency and would like this
> > > change to be made,
> > >       I would like to recruit 1 or 2 people to help me deliver this,
> > since
> > > I'm not a automake nor CMake expert.
> >
> >
>

Re: Separate Compilation of Tests

[Michael Park <mp...@apache.org>]

On Wed, Oct 5, 2016 at 9:22 AM, Joris Van Remoortere <jo...@mesosphere.io>
wrote:

> @mpark
> Can you clarify if your goal is to:
> 1) refactor the source code (maybe including things like cluster.hpp) so
> that general compilation of any file is faster. Examples include allowing
> non-header-only stout, splitting out some files, moving some more stuff to
> cpps.
> or 2) splitting out the files in the cmake / autoconf so that it's possible
> to compile a single test file?


In this thread I was trying to capture (2).

Although 2) is probably the easiest way to improved performance I was under
> the impression there were some arguments ~2 years ago around wanting to
> have all the tests in a single binary.
>

Mm... okay, I don't seem to recall :(


> I think any effort spent 1) will be valuable regardless. We've talked about
> a bunch of tasks in #1 and just needed time / commitment to them.
>

Yeah, I agree.

Some people have also suggested improved tooling. For example the gold
> linker.
>

@benjamin: I remember you were working on the gold linker stuff. As far as
I remember, it didn't get committed.
Are you interested in reviving that stuff?


> You should also see if you are ever IO bound. Debug builds generate
> binaries so large that you actually can become IO bound. There are ways to
> get around that too. There have been times where an optimized build has
> been faster than a debug build.
>

Yeah.. as far as I know optimized build builds faster. But I haven't
confirmed that.

Joris
>
> —
> *Joris Van Remoortere*
> Mesosphere
>
> On Tue, Sep 27, 2016 at 6:25 PM, Michael Park <mp...@apache.org> wrote:
>
> > On Tue, Sep 27, 2016 at 12:21 PM, Alexander Rojas <
> alexander@mesosphere.io
> > >
> > wrote:
> >
> > > The workaround I use is to go to the folder of the makefile root
> > > directory, so for
> > > Mesos, and then use the specific target, for example:
> > >
> > > For mesos-tests
> > > cd ${MESOS_SRC_ROOT}/build/src
> > > make -j4 mesos-tests && ./mesos-tests
> > >
> > > for libprocess
> > > cd ${MESOS_SRC_ROOT}/build/3rdparty/libprocess
> > > make -j4 libprocess-tests && ./libprocess-tests
> > >
> > > for stout
> > > cd ${MESOS_SRC_ROOT}/build/3rdparty/stout
> > > make -j4 stout-tests && ./stout-tests
> > >
> >
> > Alexander, I'm aware of these targets, and I guess my claim is that these
> > are not fine-grained enough.
> > Yes, I can work on a stout change and only run stout tests to iterate
> > without compiling/running all of the mesos tests.
> >
> > But I'm talking about times when I'm making mesos changes (let's say
> > involving a stout change),
> > I don't want to have to recompile basically all of the tests per
> iteration.
> >
> >
> > > I don’t think it would be hard to add top level targets that just do
> that
> > > from the top level.
> > >
> >
> > I imagine it's probably not either, but it's not what I'm looking for in
> > this thread.
> >
> > I however would like to have a target that compiles but not runs the
> tests.
> > >
> >
> > What do you mean by this? Are you looking for `make tests`?
> > As far as I'm aware, you can do `make tests`, and `make check` at every
> > level,
> > as opposed to the commands you're running to achieve what seems like the
> > same thing to me.
> >
> >
> > > > On 25 Sep 2016, at 15:56, Michael Park <mp...@apache.org> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I would like to propose a change in our build to help us improve
> > > developer
> > > > efficiency.
> > > > The proposal is to support separate compilation of unit tests.
> > > >
> > > > Currently, we have the old approach of invoking `make check -j N
> > > > GTEST_FILTER=""`, or a newer option of doing `make tests -j N`. From
> > what
> > > > I've heard the latter is slightly faster. However, when someone is
> > > > developing a specific feature, it's likely that they would like to
> make
> > > > changes and iterate on a single test file. In this workflow, having
> to
> > > > compile (virtually) __all__ of the tests is very burdensome. This
> > > situation
> > > > is not so bad if you're working in a very isolated part of the
> > codebase,
> > > > but it gets to be pretty bad if you're experimenting with parts that
> > are
> > > > widely used.
> > > >
> > > > An example of a workflow I'm aiming for would look something like:
> > > >
> > > >   1. write some code...
> > > >   2. `make check master_tests`  // compile and test
> > > >   `src/tests/master_tests.cpp`
> > > >   3. fix compilation errors...
> > > >   4. `make check master_tests`  // compile and test
> > > >   `src/tests/master_tests.cpp`
> > > >   5. change some stuff...
> > > >   6. `make check master_tests`  // compile and test
> > > >   `src/tests/master_tests.cpp`
> > > >   7. debug...
> > > >   8. `make check master_tests`  // compile and test
> > > >   `src/tests/master_tests.cpp`
> > > >   9. alright, looks good. `make check`
> > > >
> > > > I have 0 attachment to the `make check master_tests` syntax. It'll
> be a
> > > > different syntax for CMake anyways. I just think that the ability to
> > > > perform separate compilation of tests will be immensely useful.
> > > >
> > > > Some numbers to justify what I'm proposing:
> > > >
> > > >   - `make -j 8` on my laptop takes roughly 10 mins.
> > > >   - `make tests -j 8` takes about 30 mins.
> > > >
> > > > Of course, not every change you make triggers all of the tests to
> > > > recompile. But if you change components that are widely used, it does
> > end
> > > > up recompiling virtually everything. Under these circumstances, I
> would
> > > > love for each iteration to be 11 mins (10 mins + __at most__ 1 min to
> > > > compile the single test), rather than 30 mins.
> > > >
> > > > NOTE: My laptop is expensive... some people even use machines with 64
> > > cores
> > > > or whatever to compile Mesos. Not everyone has access to this kind of
> > > > equipment. We should strive for something better than "throw more
> money
> > > at
> > > > it".
> > > >
> > > > The goal of this thread for me is the following:
> > > >  (1) Capture whether this is something most people experience, or
> > whether
> > > > I'm just doing it wrong
> > > >  (2) If most people do experience this inefficiency and would like
> this
> > > > change to be made,
> > > >       I would like to recruit 1 or 2 people to help me deliver this,
> > > since
> > > > I'm not a automake nor CMake expert.
> > >
> > >
> >
>