Jenkins build is back to normal : Mahout-Quality #2128

[Apache Jenkins Server <je...@builds.apache.org>]

See <https://builds.apache.org/job/Mahout-Quality/2128/changes>

Re: Jenkins build is back to normal : Mahout-Quality #2128

[Stevo Slavić <ss...@gmail.com>]

Build still seems to be failing (see
https://builds.apache.org/job/mahout-nightly/1286/consoleFull ), maybe just
less frequently.
Will have to look into this, after release.

Kind regards,
Stevo Slavić.


On Sun, Jul 7, 2013 at 11:02 PM, Grant Ingersoll <gs...@apache.org>wrote:

>
> On Jul 6, 2013, at 4:38 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > What did the trick (as of r1500216) for last two builds to be successful
> > was serializing unit tests. At least some of them it seems are not
> designed
> > to run in parallel (they very likely share some state), and they were
> > running in parallel (1.5 per CPU core of Jenkins node on which build is
> > running), causing each other to fail randomly. Now it's all sequential.
>
> So, we undid the parallel builds?  Do you have a sense of the ones that
> were causing problems?
>
> -G

Re: Jenkins build is back to normal : Mahout-Quality #2128

[Grant Ingersoll <gs...@apache.org>]

On Jul 6, 2013, at 4:38 PM, Stevo Slavić <ss...@gmail.com> wrote:

> What did the trick (as of r1500216) for last two builds to be successful
> was serializing unit tests. At least some of them it seems are not designed
> to run in parallel (they very likely share some state), and they were
> running in parallel (1.5 per CPU core of Jenkins node on which build is
> running), causing each other to fail randomly. Now it's all sequential.

So, we undid the parallel builds?  Do you have a sense of the ones that were causing problems?

-G

Re: Jenkins build is back to normal : Mahout-Quality #2128

[Stevo Slavić <ss...@gmail.com>]

Hello team,

Success of that particular build was pure luck. Point of the commit that
triggered the build and several previous commits was to get source code
static analysis checks to work and consistently use custom settings from
buildtools module. E.g. SE_BAD_FIELD rule/check is excluded from findbugs
checks. Before these changes findbugs maven report on Jenkins used to list
that as high prio findbugs
issue<https://builds.apache.org/job/Mahout-Quality/2118/findbugsResult/HIGH/module.-1810137467/package.-1319752439/>in
modules like mahout-core, now
it doesn't<https://builds.apache.org/job/Mahout-Quality/2131/findbugsResult/HIGH/>.
I wanted to work on eliminating those quality issues, and found lack of
consistency between Jenkins and IDE annoying enough to go and fix it before
0.8 release, even while on vacation - island of Thassos, Greece, it's nice
here btw, you should all come :-)

What did the trick (as of r1500216) for last two builds to be successful
was serializing unit tests. At least some of them it seems are not designed
to run in parallel (they very likely share some state), and they were
running in parallel (1.5 per CPU core of Jenkins node on which build is
running), causing each other to fail randomly. Now it's all sequential.
Good thing is, that all Mahout build jobs are passing now. Downside is that
build time has increased. It tripled for these two:
https://builds.apache.org/job/Mahout-Quality/buildTimeTrend
https://builds.apache.org/job/Mahout-Trunk/buildTimeTrend

and nightly build took ~7x more time to complete:
https://builds.apache.org/job/mahout-nightly/buildTimeTrend

Mahout-Examples build jobs build times it seems didn't change.

Fast feedback from unit tests IMO is very important, so we should improve
the build time, better sooner than later, through more build script
configuration, but also designing unit tests for parallel execution in mind.

Kind regards,
Stevo Slavić.

On Sat, Jul 6, 2013 at 2:37 AM, Suneel Marthi <su...@yahoo.com>wrote:

> Thanks Stevo.  This could be a dumb question given my complete lack of
> understanding of all things Maven - so what was it that was causing random
> test failures?
>
>
>
>
> ________________________________
>  From: Ted Dunning <te...@gmail.com>
> To: Mahout Dev List <de...@mahout.apache.org>
> Sent: Friday, July 5, 2013 8:22 PM
> Subject: Re: Jenkins build is back to normal : Mahout-Quality #2128
>
>
> Thank you Stevo!
>
>
> On Fri, Jul 5, 2013 at 4:45 PM, Apache Jenkins Server <
> jenkins@builds.apache.org> wrote:
>
> > See <https://builds.apache.org/job/Mahout-Quality/2128/changes>
> >
> >
>

Re: Jenkins build is back to normal : Mahout-Quality #2128

[Suneel Marthi <su...@yahoo.com>]

Thanks Stevo.  This could be a dumb question given my complete lack of understanding of all things Maven - so what was it that was causing random test failures?




________________________________
 From: Ted Dunning <te...@gmail.com>
To: Mahout Dev List <de...@mahout.apache.org> 
Sent: Friday, July 5, 2013 8:22 PM
Subject: Re: Jenkins build is back to normal : Mahout-Quality #2128
 

Thank you Stevo!


On Fri, Jul 5, 2013 at 4:45 PM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> See <https://builds.apache.org/job/Mahout-Quality/2128/changes>
>
>

Re: Jenkins build is back to normal : Mahout-Quality #2128

[Ted Dunning <te...@gmail.com>]

Thank you Stevo!


On Fri, Jul 5, 2013 at 4:45 PM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> See <https://builds.apache.org/job/Mahout-Quality/2128/changes>
>
>