Re: Parallel test runner added

[Alex Rukletsov <al...@mesosphere.com>]

This is great, Benjamin!

I've used it the whole day today and it is awesome. (It will become
insanely great once MESOS-6387 is resolved.)

Thanks for everyone who made this happen, also on behalf of my employer : )

Alex.

On Thu, Oct 13, 2016 at 11:28 PM, Benjamin Bannier <
benjamin.bannier@mesosphere.io> wrote:

>
> Hi,
>
> Since most tests in the Mesos, libprocess, and stout test suites can
> be executed in parallel (the exception being some `ROOT` tests with
> global side effects in Mesos), we recently added a parallel test
> runner `support/mesos-gtest-runner.py`. This should allow to
> potentially significantly speed up running of test suites.
>
> To enable automatic parallel execution of tests for test targets
> executed during `make check`, configure Mesos with the option
> `--enable-parallel-test-execution`. This will configure the test runner
> to run all tests but the `ROOT` tests in parallel; `ROOT` tests will
> be run in a separate, sequential step.
>
> * * *
>
> We use the environment variable `TEST_DRIVER` to drive parallel test
> execution. By setting this variable to an empty string you can
> temporarily disable configured parallel execution, e.g.,
>
>     % make check TEST_DRIVER=
>
> By setting this environment variable you have control over the test
> runner itself and its arguments, even without enabling parallel test
> during `./configure` time. Be aware that many `ROOT` tests cannot be
> run in parallel.
>
>
> The current settings oversubscribe the machine by running `#cores*1.5`
> parallel jobs. This was driven by the observation that currently our
> tests by and large do not make extended use of even a single core.
> The number of parallel jobs can by controlled with the `-j` flag of
> the test runner.
>
> Since making more use of the machine will likely increase machine load
> during test execution, running tests in parallel might expose test
> flakiness. Tests might also fail to run in parallel if testcases e.g.,
> write data to hardcoded locations or use hardcoded ports. Please file
> JIRA tickets for such tests if they do not yet exist.
>
>
> There is still some work needed to improve reporting from parallel
> tests. We currently use a very silent mode if tests are running
> without failures, and just report the logs of failed jobs in case of
> failure. MESOS-6387 sketches out possible future improvements in this
> area.
>
>
> Happy testing,
>
> Benjamin with help from Kevin & Till
>
>