You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2015/11/03 08:03:06 UTC

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

I just did an edit of all of our jenkins configuration. I changed the
post-build zombie section. The surefire-killer was hanging out here it
seems (For detail, see HBASE-14589 Looking for the surefire-killer; builds
being killed...). I may have messed up builds. Will fix in morning if so
(currently builds seem a bit backed up so won't wait around).

St.Ack

On Fri, Oct 30, 2015 at 9:18 AM, Sean Busbey <bu...@cloudera.com> wrote:

> Hi Folks,
>
> Brief summary of cleanup/changes in our builds. ref thread '[DISCUSS]
> reducing our load on builds.a.o' on reasoning or if you have concerns.
>
> Our project job view now has a brief summary of what's going on:
>
> https://builds.apache.org/view/H-L/view/HBase/
>
> * Where we had matrix builds, I confirmed they're doing the same thing
> (in at least one test-configuration) as the non-matrix version
> * I disabled jobs that were duplicating work done in matrix jobs, or
> that only worked on a tag
> * Made sure all non-disabled 0.98+ jobs are set to notify jira +
> builds mailing list
> * I updated pre-1.1 jobs to look for changes 1/day
> * I updated 1.1 jobs to look for changes every 8 hours
> * I updated 1.2, 1.3, and trunk to look for changes every 4 hours
>
> There's still some non-pressing clean up work to do, so I'll file a
> jira later today just to make sure it's all recorded somewhere.
>
> On Mon, Jun 15, 2015 at 11:55 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > Added new HBase-1.3 build or branch-1 and changed the existing 1.2
> builds to
> > use branch-1.2. (ref HBASE-13912)
> >
> > On Sun, Jun 14, 2015 at 1:11 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>
> >> I've add a new build test to run the ITs under sunny day conditions
> using
> >> minicluster for both java 7 and java 8 on the 1.2 line.
> >>
> >> https://builds.apache.org/job/HBase-1.2-IT/
> >>
> >> I haven't turned on notifications yet, because BulkIngest and TestIngest
> >> are read. details on HBASE-13750
> >>
> >> On Sat, Jun 13, 2015 at 8:28 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>>
> >>> sigh. that should have been "is now a matrix build".
> >>>
> >>> On Sat, Jun 13, 2015 at 7:31 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>>>
> >>>> The HBase-1.2 build[1] is not a matrix build that runs JDK7 and JDK8
> in
> >>>> parallel.
> >>>>
> >>>> I saved the old job for now and left it disabled[2].
> >>>>
> >>>>
> >>>> [1]: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/
> >>>> [2]:
> >>>>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2%20non-matrix/
> >>>>
> >>>> On Fri, Apr 10, 2015 at 3:42 PM, Sean Busbey <bu...@cloudera.com>
> >>>> wrote:
> >>>>>
> >>>>> Sorry for the noise. I also updated the checkstyle step on
> HBase-TRUNK
> >>>>>
> >>>>> from
> >>>>>     mvn checkstyle:checkstyle
> >>>>> to
> >>>>>     mvn -DskipTests package checkstyle:checkstyle
> >>>>>
> >>>>> to deal with the same issue. Looks all clear now.
> >>>>>
> >>>>> On Fri, Apr 10, 2015 at 3:03 PM, Sean Busbey <bu...@cloudera.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Updated the following builds:
> >>>>>>
> >>>>>> * HBase-TRUNK
> >>>>>>
> >>>>>> moved from
> >>>>>>
> >>>>>>     mvn clean compile findbugs:findbugs
> >>>>>>
> >>>>>> to
> >>>>>>
> >>>>>>    mvn clean -DskipTests package findbugs:findbugs
> >>>>>>
> >>>>>> To work around the known issue where we can't do a bootstrap compile
> >>>>>> without previous install or remote SNAPSHOT artifacts. (recently
> triggered
> >>>>>> by the addition of the procedure module on master)
> >>>>>>
> >>>>>> On Mon, Mar 30, 2015 at 8:49 PM, 张铎 <pa...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Make findbugs and checkstyle plugins always run.
> >>>>>>> The default behavior only publishes result for stable builds, but
> >>>>>>> master is
> >>>>>>> not always stable and sometimes we introduce new warnings in
> unstable
> >>>>>>> builds(see builds 6310-6314).
> >>>>>>> And findbugs and checkstyle will not fail unless we abort the
> >>>>>>> building
> >>>>>>> process I think, so it is safe to turn it on every time.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> 2015-03-22 22:44 GMT+08:00 Ted Yu <yu...@gmail.com>:
> >>>>>>>
> >>>>>>> > +1 on letting HBase-TRUNK jenkins show coverage report.
> >>>>>>> >
> >>>>>>> > Cheers
> >>>>>>> >
> >>>>>>> > On Sun, Mar 22, 2015 at 5:51 AM, 张铎 <pa...@gmail.com>
> wrote:
> >>>>>>> >
> >>>>>>> > > Going to change the config of HBase-TRUNK jenkins to show
> >>>>>>> > > findbugs,
> >>>>>>> > > checkstyle and jacoco coverage report.
> >>>>>>> > > The config has been tested on
> >>>>>>> > > https://builds.apache.org/job/HBase-TRUNK-jacoco for nearly 30
> >>>>>>> > > times.
> >>>>>>> > Not
> >>>>>>> > > much different from HBase-TRUNK unless it runs ~30% slower(the
> >>>>>>> > > overhead
> >>>>>>> > of
> >>>>>>> > > collecting information for code coverage).
> >>>>>>> > > Thanks.
> >>>>>>> > >
> >>>>>>> > > 2015-03-12 5:08 GMT+08:00 Andrew Purtell <apurtell@apache.org
> >:
> >>>>>>> > >
> >>>>>>> > > > +1, thanks a lot for improving our build hygiene.
> >>>>>>> > > >
> >>>>>>> > > > On Wed, Mar 11, 2015 at 2:01 PM, Enis Söztutar
> >>>>>>> > > > <en...@gmail.com>
> >>>>>>> > > wrote:
> >>>>>>> > > >
> >>>>>>> > > > > Thanks Sean. This is great.
> >>>>>>> > > > >
> >>>>>>> > > > > Enis
> >>>>>>> > > > >
> >>>>>>> > > > > On Wed, Mar 11, 2015 at 1:54 PM, Sean Busbey
> >>>>>>> > > > > <bu...@cloudera.com>
> >>>>>>> > > > wrote:
> >>>>>>> > > > >
> >>>>>>> > > > > > FYI I just finished chasing down the breakage for mvn
> site
> >>>>>>> > > > > > on all
> >>>>>>> > > patch
> >>>>>>> > > > > > builds.
> >>>>>>> > > > > >
> >>>>>>> > > > > > HBASE-13191 consolidates the few places in the test-patch
> >>>>>>> > > > > > code
> >>>>>>> > where
> >>>>>>> > > we
> >>>>>>> > > > > > hard coded MAVEN_OPTS.
> >>>>>>> > > > > >
> >>>>>>> > > > > > If you look at the PreCommit job now, we use the "set
> >>>>>>> > > > > > environment
> >>>>>>> > > > > > variables" option to set MAVEN_OPTS and then everything
> >>>>>>> > > > > > else
> >>>>>>> > respects
> >>>>>>> > > > > that
> >>>>>>> > > > > > setting.
> >>>>>>> > > > > >
> >>>>>>> > > > > > I set the initial value to be a combination of the memory
> >>>>>>> > limitations
> >>>>>>> > > > > we've
> >>>>>>> > > > > > been actually running with (the ~6G was getting ignored)
> >>>>>>> > > > > > and the
> >>>>>>> > > > permgen
> >>>>>>> > > > > > needed for site.
> >>>>>>> > > > > >
> >>>>>>> > > > > > MAVEN_OPTS=-Xmx3100M -XX:-UsePerfData
> -XX:MaxPermSize=256m
> >>>>>>> > > > > >
> >>>>>>> > > > > > On Mon, Feb 23, 2015 at 11:46 PM, Stack <
> stack@duboce.net>
> >>>>>>> > > > > > wrote:
> >>>>>>> > > > > >
> >>>>>>> > > > > > > I just made TRUNK and branch-1 builds use same jvm as
> >>>>>>> > > > > > > patch-build
> >>>>>>> > > > > > > (hadoopqa) -- i.e. jdku51 -- and I set the MAVEN_OPTS
> to
> >>>>>>> > > > > > > be the
> >>>>>>> > > same
> >>>>>>> > > > as
> >>>>>>> > > > > > > those of trunk build too, setting
> >>>>>>> > > > > > > MAVEN_OPTS="-Xmx6100m"... it
> >>>>>>> > had
> >>>>>>> > > > been
> >>>>>>> > > > > > > 3000.
> >>>>>>> > > > > > >
> >>>>>>> > > > > > > Yours,
> >>>>>>> > > > > > > St.Ack
> >>>>>>> > > > > > >
> >>>>>>> > > > > > > On Wed, Dec 31, 2014 at 4:22 PM, Stack <
> stack@duboce.net>
> >>>>>>> > > > > > > wrote:
> >>>>>>> > > > > > >
> >>>>>>> > > > > > > > I upped hadoopqa retention to keep last 100 builds
> and
> >>>>>>> > > > > > > > or last
> >>>>>>> > 7
> >>>>>>> > > > > days,
> >>>>>>> > > > > > > > whichever comes first.
> >>>>>>> > > > > > > > St.Ack
> >>>>>>> > > > > > > >
> >>>>>>> > > > > > > > On Tue, Nov 4, 2014 at 9:38 AM, Stack
> >>>>>>> > > > > > > > <st...@duboce.net>
> >>>>>>> > wrote:
> >>>>>>> > > > > > > >
> >>>>>>> > > > > > > >> Branch-1 and master have stabilized and now run
> mostly
> >>>>>>> > > > > > > >> blue
> >>>>>>> > > (give
> >>>>>>> > > > or
> >>>>>>> > > > > > > take
> >>>>>>> > > > > > > >> the odd failure) [1][2]. Having a mostly blue
> branch-1
> >>>>>>> > > > > > > >> has
> >>>>>>> > > helped
> >>>>>>> > > > us
> >>>>>>> > > > > > > >> identify at least one destabilizing commit in the
> last
> >>>>>>> > > > > > > >> few
> >>>>>>> > days,
> >>>>>>> > > > > maybe
> >>>>>>> > > > > > > two;
> >>>>>>> > > > > > > >> this is as it should be (smile).
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> Lets keep our builds blue. If you commit a patch,
> make
> >>>>>>> > > > > > > >> sure
> >>>>>>> > > > > subsequent
> >>>>>>> > > > > > > >> builds stay blue. You can subscribe to
> >>>>>>> > builds@hbase.apache.org
> >>>>>>> > > to
> >>>>>>> > > > > get
> >>>>>>> > > > > > > >> notice of failures if not already subscribed.
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> Thanks,
> >>>>>>> > > > > > > >> St.Ack
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> 1.
> >>>>>>> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.0/
> >>>>>>> > > > > > > >> 2.
> >>>>>>> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-TRUNK/
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >> On Mon, Oct 13, 2014 at 4:41 PM, Stack
> >>>>>>> > > > > > > >> <st...@duboce.net>
> >>>>>>> > > wrote:
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >>> A few notes on testing.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Too long to read, infra is more capable now and
> after
> >>>>>>> > > > > > > >>> some
> >>>>>>> > > work,
> >>>>>>> > > > we
> >>>>>>> > > > > > are
> >>>>>>> > > > > > > >>> seeing branch-1 and trunk mostly running blue. Lets
> >>>>>>> > > > > > > >>> try and
> >>>>>>> > > keep
> >>>>>>> > > > it
> >>>>>>> > > > > > > this
> >>>>>>> > > > > > > >>> way going forward.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Apache Infra has new, more capable hardware.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> A recent spurt of test fixing combined with more
> >>>>>>> > > > > > > >>> capable
> >>>>>>> > > hardware
> >>>>>>> > > > > > seems
> >>>>>>> > > > > > > >>> to have gotten us to a new place; tests are mostly
> >>>>>>> > > > > > > >>> passing
> >>>>>>> > now
> >>>>>>> > > on
> >>>>>>> > > > > > > branch-1
> >>>>>>> > > > > > > >>> and master.  Lets try and keep it this way and
> start
> >>>>>>> > > > > > > >>> to trust
> >>>>>>> > > our
> >>>>>>> > > > > > test
> >>>>>>> > > > > > > runs
> >>>>>>> > > > > > > >>> again.  Just a few flakies remain.  Lets try and
> nail
> >>>>>>> > > > > > > >>> them.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Our tests now run in parallel with other test
> suites
> >>>>>>> > > > > > > >>> where
> >>>>>>> > > > previous
> >>>>>>> > > > > > we
> >>>>>>> > > > > > > >>> ran alone. You can see this sometimes when our
> zombie
> >>>>>>> > detector
> >>>>>>> > > > > > reports
> >>>>>>> > > > > > > >>> tests from another project altogether as lingerers
> >>>>>>> > > > > > > >>> (To be
> >>>>>>> > > fixed).
> >>>>>>> > > > > > > Some of
> >>>>>>> > > > > > > >>> our tests are failing because a concurrent hbase
> run
> >>>>>>> > > > > > > >>> is
> >>>>>>> > undoing
> >>>>>>> > > > > > > classes and
> >>>>>>> > > > > > > >>> data from under it. Also, lets fix.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Our tests are brittle. It takes 75minutes for them
> to
> >>>>>>> > complete.
> >>>>>>> > > > > Many
> >>>>>>> > > > > > > >>> are heavy-duty integration tests starting up
> multiple
> >>>>>>> > clusters
> >>>>>>> > > > and
> >>>>>>> > > > > > > >>> mapreduce all in the one JVM. It is a miracle they
> >>>>>>> > > > > > > >>> pass at
> >>>>>>> > all.
> >>>>>>> > > > > > > Usually
> >>>>>>> > > > > > > >>> integration tests have been cast as unit tests
> >>>>>>> > > > > > > >>> because there
> >>>>>>> > > was
> >>>>>>> > > > no
> >>>>>>> > > > > > > where
> >>>>>>> > > > > > > >>> else for them to get an airing.  We have the
> hbase-it
> >>>>>>> > > > > > > >>> suite
> >>>>>>> > now
> >>>>>>> > > > > which
> >>>>>>> > > > > > > would
> >>>>>>> > > > > > > >>> be a more apt place but until these are run on a
> >>>>>>> > > > > > > >>> regular
> >>>>>>> > basis
> >>>>>>> > > in
> >>>>>>> > > > > > > public
> >>>>>>> > > > > > > >>> for all to see, the fat integration tests disguised
> >>>>>>> > > > > > > >>> as unit
> >>>>>>> > > tests
> >>>>>>> > > > > > will
> >>>>>>> > > > > > > >>> remain.  A review of our current unit tests weeding
> >>>>>>> > > > > > > >>> the old
> >>>>>>> > > cruft
> >>>>>>> > > > > and
> >>>>>>> > > > > > > the
> >>>>>>> > > > > > > >>> no longer relevant or duplicates would be a nice
> >>>>>>> > > > > > > >>> undertaking
> >>>>>>> > if
> >>>>>>> > > > > > > someone is
> >>>>>>> > > > > > > >>> looking to contribute.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> Alex Newman has been working on making our tests
> work
> >>>>>>> > > > > > > >>> up on
> >>>>>>> > > > travis
> >>>>>>> > > > > > and
> >>>>>>> > > > > > > >>> circle-ci.  That'll be sweet when it goes
> end-to-end.
> >>>>>>> > > > > > > >>> He
> >>>>>>> > also
> >>>>>>> > > > > added
> >>>>>>> > > > > > in
> >>>>>>> > > > > > > >>> some "type" categorizations -- client, filter,
> >>>>>>> > > > > > > >>> mapreduce --
> >>>>>>> > > > > alongside
> >>>>>>> > > > > > > our
> >>>>>>> > > > > > > >>> old "sizing" categorizations of small/medium/large.
> >>>>>>> > > > > > > >>> His
> >>>>>>> > > thinking
> >>>>>>> > > > > is
> >>>>>>> > > > > > > that
> >>>>>>> > > > > > > >>> we can run these categorizations in parallel so we
> >>>>>>> > > > > > > >>> could run
> >>>>>>> > > the
> >>>>>>> > > > > > total
> >>>>>>> > > > > > > >>> suite in about the time of the longest test, say
> >>>>>>> > 20-30minutes?
> >>>>>>> > > > We
> >>>>>>> > > > > > > could
> >>>>>>> > > > > > > >>> even change Apache to run them this way.
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>> FYI,
> >>>>>>> > > > > > > >>> St.Ack
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>>
> >>>>>>> > > > > > > >>
> >>>>>>> > > > > > > >
> >>>>>>> > > > > > >
> >>>>>>> > > > > >
> >>>>>>> > > > > >
> >>>>>>> > > > > >
> >>>>>>> > > > > > --
> >>>>>>> > > > > > Sean
> >>>>>>> > > > > >
> >>>>>>> > > > >
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > >
> >>>>>>> > > > --
> >>>>>>> > > > Best regards,
> >>>>>>> > > >
> >>>>>>> > > >    - Andy
> >>>>>>> > > >
> >>>>>>> > > > Problems worthy of attack prove their worth by hitting back.
> -
> >>>>>>> > > > Piet
> >>>>>>> > Hein
> >>>>>>> > > > (via Tom White)
> >>>>>>> > > >
> >>>>>>> > >
> >>>>>>> >
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Sean
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Sean
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sean
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Sean
> >>
> >>
> >>
> >>
> >> --
> >> Sean
> >
> >
> >
> >
> > --
> > Sean
>
>
>
> --
> Sean
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

Posted by stack <sa...@gmail.com>.

Thank you Sean (and Andrew)
St.Ack
On Jan 22, 2016 10:12 PM, "Sean Busbey" <bu...@apache.org> wrote:

> Andrew in infra made us a label that covers all the hosts save H2.
> I've updated all the nightly builds to use it.
>
> (specifying a label expression as we do in precommit doesn't work on
> matrix builds because the & and | from the expression end up in the
> filesystem path)
>
> On Fri, Jan 22, 2016 at 3:08 PM, Sean Busbey <bu...@cloudera.com> wrote:
> > we should probably ensure the earlier branch builds also exclude H2.
> >
> > I'll leave myself a note to look at it this evening. If anyone gets to it
> > before then, please update here.
> >
> > On Fri, Jan 22, 2016 at 1:33 PM, Stack <st...@duboce.net> wrote:
> >
> >> Related to the below, I just changed the trunk matrix build job to
> exclude
> >> H2 from the build roster (with Sean's help); it seems to be responsible
> for
> >> this failure type -- *Caused by: java.lang.IndexOutOfBoundsException:
> >> Index: 0, Size: 0* -- in particular. Here is recent example:
> >>
> >>
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/652/jdk=latest1.7,label=Hadoop/console
> >>
> >> Lets see if it helps.
> >>
> >> While I have your attention, see the nice checkstyle and findbug graph
> >> trajectories here:
> >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/
> >>
> >> St.Ack
> >>
> >>
> >> On Tue, Jan 19, 2016 at 7:02 PM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >>
> >> > On Tue, Jan 19, 2016 at 11:48 AM, Stack <st...@duboce.net> wrote:
> >> >
> >> > > On Tue, Jan 19, 2016 at 5:46 AM, Sean Busbey <bu...@apache.org>
> >> wrote:
> >> > >
> >> > > > We could start forcing a clean repository on every build (though
> this
> >> > > > seems heavy handed).
> >> > > >
> >> > > > IIRC, we ran into this ages ago and it was one particular
> dependency.
> >> > > > Presuming we can track down what that was, we could add some
> >> pre-build
> >> > > > work that verifies a known good copy of that dependency is
> present.
> >> > > >
> >> > > > For now, I've blacklisted the H2 build host again, since that's
> the
> >> > > > only host this has been happening on. We added it back last week
> so
> >> > > > that Jon could test a fix from infra.
> >> > > >
> >> > > >
> >> > > Thanks Sean.
> >> > >
> >> > > Yeah, clean repo each time would be OTT.
> >> > >
> >> > > If you have pointers on how to find the bad dependency, I'll dig.
> >> > >
> >> > > Of course, all builds fine locally.
> >> > >
> >> > > St.Ack
> >> > >
> >> > >
> >> > >
> >> > Relevant bits from our old precommit build (lucky we kept it! ;) )
> >> >
> >> >
> >> > ----
> >> > ...
> >> > # holding place for local repo
> >> > if [ -d "${WORKSPACE}/maven_repo" ]; then
> >> >   echo "removing hbase artifacts from prior runs."
> >> >   rm -rf "${WORKSPACE}/maven_repo/org/apache/hbase"
> >> >   echo "removing javax.inject in case we got a bad dependency"
> >> >   rm -rf "${WORKSPACE}/maven_repo/javax/inject"
> >> > # uncomment to list entire contents of repo
> >> > #  find "${WORKSPACE}/maven_repo"
> >> > else
> >> >   mkdir "${WORKSPACE}/maven_repo"
> >> > fi
> >> >
> >> > ...
> >> >
> >> > if [ "$?" -ne "0" ]; then
> >> >   echo "test patch failed, checking javax.inject dependency"
> >> >   find "${WORKSPACE}/maven_repo/javax/inject"
> >> >   cat
> >> >
> "${WORKSPACE}/maven_repo/javax/inject/javax.inject/1/javax.inject-1.pom"
> >> >   exit 1
> >> > fi
> >> > ---
> >> >
> >> > So it looks like javax:inject was the culprit before. A first step
> would
> >> be
> >> > to remove it from our local repos before running; that'll require
> looking
> >> > up how Yetus names the branch-specific maven repos. A better step
> would
> >> be
> >> > to manually install it from a known good location so we can avoid
> >> whatever
> >> > nonsense is happening on H2.
> >> >
> >>
> >
> >
> >
> > --
> > Sean
>

Re: Testing and CI -- Apache Jenkins Builds (WAS -> Re: Testing)

Posted by Sean Busbey <bu...@apache.org>.

Andrew in infra made us a label that covers all the hosts save H2.
I've updated all the nightly builds to use it.

(specifying a label expression as we do in precommit doesn't work on
matrix builds because the & and | from the expression end up in the
filesystem path)

On Fri, Jan 22, 2016 at 3:08 PM, Sean Busbey <bu...@cloudera.com> wrote:
> we should probably ensure the earlier branch builds also exclude H2.
>
> I'll leave myself a note to look at it this evening. If anyone gets to it
> before then, please update here.
>
> On Fri, Jan 22, 2016 at 1:33 PM, Stack <st...@duboce.net> wrote:
>
>> Related to the below, I just changed the trunk matrix build job to exclude
>> H2 from the build roster (with Sean's help); it seems to be responsible for
>> this failure type -- *Caused by: java.lang.IndexOutOfBoundsException:
>> Index: 0, Size: 0* -- in particular. Here is recent example:
>>
>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/652/jdk=latest1.7,label=Hadoop/console
>>
>> Lets see if it helps.
>>
>> While I have your attention, see the nice checkstyle and findbug graph
>> trajectories here:
>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Trunk_matrix/
>>
>> St.Ack
>>
>>
>> On Tue, Jan 19, 2016 at 7:02 PM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>> > On Tue, Jan 19, 2016 at 11:48 AM, Stack <st...@duboce.net> wrote:
>> >
>> > > On Tue, Jan 19, 2016 at 5:46 AM, Sean Busbey <bu...@apache.org>
>> wrote:
>> > >
>> > > > We could start forcing a clean repository on every build (though this
>> > > > seems heavy handed).
>> > > >
>> > > > IIRC, we ran into this ages ago and it was one particular dependency.
>> > > > Presuming we can track down what that was, we could add some
>> pre-build
>> > > > work that verifies a known good copy of that dependency is present.
>> > > >
>> > > > For now, I've blacklisted the H2 build host again, since that's the
>> > > > only host this has been happening on. We added it back last week so
>> > > > that Jon could test a fix from infra.
>> > > >
>> > > >
>> > > Thanks Sean.
>> > >
>> > > Yeah, clean repo each time would be OTT.
>> > >
>> > > If you have pointers on how to find the bad dependency, I'll dig.
>> > >
>> > > Of course, all builds fine locally.
>> > >
>> > > St.Ack
>> > >
>> > >
>> > >
>> > Relevant bits from our old precommit build (lucky we kept it! ;) )
>> >
>> >
>> > ----
>> > ...
>> > # holding place for local repo
>> > if [ -d "${WORKSPACE}/maven_repo" ]; then
>> >   echo "removing hbase artifacts from prior runs."
>> >   rm -rf "${WORKSPACE}/maven_repo/org/apache/hbase"
>> >   echo "removing javax.inject in case we got a bad dependency"
>> >   rm -rf "${WORKSPACE}/maven_repo/javax/inject"
>> > # uncomment to list entire contents of repo
>> > #  find "${WORKSPACE}/maven_repo"
>> > else
>> >   mkdir "${WORKSPACE}/maven_repo"
>> > fi
>> >
>> > ...
>> >
>> > if [ "$?" -ne "0" ]; then
>> >   echo "test patch failed, checking javax.inject dependency"
>> >   find "${WORKSPACE}/maven_repo/javax/inject"
>> >   cat
>> > "${WORKSPACE}/maven_repo/javax/inject/javax.inject/1/javax.inject-1.pom"
>> >   exit 1
>> > fi
>> > ---
>> >
>> > So it looks like javax:inject was the culprit before. A first step would
>> be
>> > to remove it from our local repos before running; that'll require looking
>> > up how Yetus names the branch-specific maven repos. A better step would
>> be
>> > to manually install it from a known good location so we can avoid
>> whatever
>> > nonsense is happening on H2.
>> >
>>
>
>
>
> --
> Sean