You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Stack <st...@duboce.net> on 2015/08/20 02:05:55 UTC
Re: Testing and CI -- Apache Jenkins Builds
Trunk builds are failing with OOME, can't create native thread (HBASE-14262)
Anyone know of any recent INFRA changes?
Imperfect triage has our problem starting in build #15134, Aug 18, 2015
5:10:02 AM
Otherwise, was going to mess with heap sizings and trying to make the big
tests smaller so we can get trunk passing again.
Thanks,
St.Ack
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
On Mon, Aug 31, 2015 at 8:01 AM, Sean Busbey <bu...@cloudera.com> wrote:
> FYI, I've changed the precommit test to run using the test-patch.sh file
> from a branch for HBASE-14337 so I can make changes needed to allow the
> workspace to have a maven repo just for our builds (rather than the shared
> one with the jenkins user).
>
> submitted patches are still applied and run against their branch (default
> master).
>
> Once I have a coherent patch I'll post it to HBASE-14337 and we can stop
> using the branch.
>
>
I just ran into this. It took a bit to figure it ("Why are my changes to
test-patch.sh not having an effect..." -- smile).
So, I think its probably a good idea to run test-patch.sh from a branch.
The alternative is checking in latest test-patch.sh changes to all branches
(test-patch.sh and pieces of this script are scattered enough already with
the zombie finding portion pasted as after-build script to run on build
completion on a bunch of our branches -- need to fix).
I added a branch to our repo called jenkins.build. It should be clearer
that this is where we do jenkins.build from (a bit clearer than HBASE-14337
-- smile). I then changed the patch build to use this branch doing jenkins
build (added comments in the script up in jenkins so hopefully it is a bit
plain what is going on).
Good on you Sean,
St.Ack
> On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
>
> > A deadlock was causing TestDistributedLogSplitting to hang. Elliott fixed
> > it over in HBASE-14274. This was the source of the zombie processes
> > reported by INFRA (at least, it matched the one instance they reported).
> >
> > Patch builds were disabled overnight (for a different reason) but are
> back
> > again.
> >
> > St.Ack
> >
> >
> >
> >
> >
> > On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
> >
> > > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> https://issues.apache.org/jira/browse/BUILDS-109
> > >>
> > >
> > > This became https://issues.apache.org/jira/browse/INFRA-10150 which
> > > becomes BUILDS-111.
> > >
> > > Errant left-over java processes sucking up resources. Trunk is
> > > particularly susceptible because we spin up way more threads than in
> > other
> > > branches (async rpc client). See HBASE-14262 for follow-up.
> > >
> > > St.Ack
> > >
> > >
> > >>
> > >> St.Ack
> > >>
> > >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
> > >>
> > >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
> > >>>
> > >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com>
> > >>>> wrote:
> > >>>>
> > >>>>> I don't recall anything, but I'll go dig through the infra@ list
> > >>>>> later this
> > >>>>> evening to make sure.
> > >>>>>
> > >>>>> Is this on all the H* build hosts?
> > >>>>>
> > >>>>>
> > >>>> Seemingly, yes.
> > >>>>
> > >>>> I can look at infra changes, no prob. Sean. Was just wondering if
> > >>>> anyone knew of a change.
> > >>>>
> > >>>>
> > >>> I don't see anything in INFRA JIRA mailing list related.
> > >>>
> > >>> Let me file an INFRA BUILDS issue and see if anything comes of it.
> > >>>
> > >>> St.Ack
> > >>>
> > >>>
> > >>>
> > >>>> Thanks,
> > >>>> St.Ack
> > >>>>
> > >>>>
> > >>>>
> > >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
> > >>>>>
> > >>>>> > Trunk builds are failing with OOME, can't create native thread
> > >>>>> > (HBASE-14262)
> > >>>>> >
> > >>>>> > Anyone know of any recent INFRA changes?
> > >>>>> >
> > >>>>> > Imperfect triage has our problem starting in build #15134, Aug
> 18,
> > >>>>> 2015
> > >>>>> > 5:10:02 AM
> > >>>>> >
> > >>>>> > Otherwise, was going to mess with heap sizings and trying to make
> > >>>>> the big
> > >>>>> > tests smaller so we can get trunk passing again.
> > >>>>> >
> > >>>>> > Thanks,
> > >>>>> > St.Ack
> > >>>>> >
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Sean
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>
>
>
> --
> Sean
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
...and Andrew Bayer has been helping out up on INFRA (see up on ticket).
Jenkins currently down for patch builds (probably while Andrew is up in the
air). It is an airy build day today since Sean also aloft.
St.Ack
On Tue, Sep 15, 2015 at 7:25 AM, Sean Busbey <bu...@cloudera.com> wrote:
> I updated the job to export the location of Maven 3.0.5 and then hard-set
> MAVEN_HOME to point at that version. For now, I've un-blacklisted H0, H4,
> H5, and H6 since we have emails saying they were removed for maven. I think
> H2 was the javax-inject dependency but I can't find a note about it or H10
> or H11 at the moment.
>
> (I also fixed a bug where we wouldn't fail the build if test-patch failed
> but we were then able to properly list the problematic javax-inject
> dependency files in our local repo.)
>
> On Tue, Sep 15, 2015 at 7:31 AM, Stack <st...@duboce.net> wrote:
>
> > Filed https://issues.apache.org/jira/browse/INFRA-10429 on missing mvn.
> > St.Ack
> >
> > On Mon, Sep 14, 2015 at 7:42 PM, Stack <st...@duboce.net> wrote:
> >
> > > Removed H4. It's forgotten where mvn is.
> > >
> > > Will try to talk to infra about getting this fixed on !H0 && !H10 &&
> !H2
> > > && !H6 && !H5 && !H4
> > >
> > > St.Ack
> > >
> > > On Thu, Sep 10, 2015 at 3:09 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> I removed H5 and H6 from rotation. They can't find mvn.
> > >> St.Ack
> > >>
> > >> On Fri, Sep 4, 2015 at 10:42 AM, Stack <st...@duboce.net> wrote:
> > >>
> > >>> You the man Sean!
> > >>>
> > >>> For those following along, this fail looks like this in the logs:
> > >>>
> > >>> "Error invoking method 'get(java.lang.Integer)' in
> java.util.ArrayList
> > >>> at META-INF/LICENSE.vm[line 1619, column 22]:
> > InvocationTargetException:
> > >>> Index: 0, Size: 0 -> [Help 1]"
> > >>> St.Ack
> > >>>
> > >>> On Fri, Sep 4, 2015 at 10:24 AM, Sean Busbey <bu...@cloudera.com>
> > >>> wrote:
> > >>>
> > >>>> Recent run found that javax.inject is the dependency that is somehow
> > >>>> getting no license locally on some build boxes. it looks fine on
> > maven.
> > >>>>
> > >>>> I've added steps to the precommit build that ensures this particular
> > >>>> dependency is pulled down each time and shows us its contents in the
> > >>>> event
> > >>>> that things fail.
> > >>>>
> > >>>> On Mon, Aug 31, 2015 at 11:17 AM, Stack <st...@duboce.net> wrote:
> > >>>>
> > >>>> > Thanks Sean. That seems to get us going again. I pushed Anoops
> > >>>> HBASE-13158
> > >>>> > and its running.
> > >>>> > St.Ack
> > >>>> >
> > >>>> > On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <busbey@cloudera.com
> >
> > >>>> wrote:
> > >>>> >
> > >>>> > > and now I've removed H0 from the precommit rotation, because of
> > >>>> repeated
> > >>>> > "I
> > >>>> > > can't find maven" errors. :/
> > >>>> > >
> > >>>> > > On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <
> > busbey@cloudera.com>
> > >>>> > wrote:
> > >>>> > >
> > >>>> > > > FYI, I've changed the precommit test to run using the
> > >>>> test-patch.sh
> > >>>> > file
> > >>>> > > > from a branch for HBASE-14337 so I can make changes needed to
> > >>>> allow the
> > >>>> > > > workspace to have a maven repo just for our builds (rather
> than
> > >>>> the
> > >>>> > > shared
> > >>>> > > > one with the jenkins user).
> > >>>> > > >
> > >>>> > > > submitted patches are still applied and run against their
> branch
> > >>>> > (default
> > >>>> > > > master).
> > >>>> > > >
> > >>>> > > > Once I have a coherent patch I'll post it to HBASE-14337 and
> we
> > >>>> can
> > >>>> > stop
> > >>>> > > > using the branch.
> > >>>> > > >
> > >>>> > > > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net>
> > wrote:
> > >>>> > > >
> > >>>> > > >> A deadlock was causing TestDistributedLogSplitting to hang.
> > >>>> Elliott
> > >>>> > > fixed
> > >>>> > > >> it over in HBASE-14274. This was the source of the zombie
> > >>>> processes
> > >>>> > > >> reported by INFRA (at least, it matched the one instance they
> > >>>> > reported).
> > >>>> > > >>
> > >>>> > > >> Patch builds were disabled overnight (for a different reason)
> > >>>> but are
> > >>>> > > back
> > >>>> > > >> again.
> > >>>> > > >>
> > >>>> > > >> St.Ack
> > >>>> > > >>
> > >>>> > > >>
> > >>>> > > >>
> > >>>> > > >>
> > >>>> > > >>
> > >>>> > > >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net>
> > wrote:
> > >>>> > > >>
> > >>>> > > >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net>
> > >>>> wrote:
> > >>>> > > >> >
> > >>>> > > >> >> https://issues.apache.org/jira/browse/BUILDS-109
> > >>>> > > >> >>
> > >>>> > > >> >
> > >>>> > > >> > This became
> > https://issues.apache.org/jira/browse/INFRA-10150
> > >>>> which
> > >>>> > > >> > becomes BUILDS-111.
> > >>>> > > >> >
> > >>>> > > >> > Errant left-over java processes sucking up resources. Trunk
> > is
> > >>>> > > >> > particularly susceptible because we spin up way more
> threads
> > >>>> than in
> > >>>> > > >> other
> > >>>> > > >> > branches (async rpc client). See HBASE-14262 for follow-up.
> > >>>> > > >> >
> > >>>> > > >> > St.Ack
> > >>>> > > >> >
> > >>>> > > >> >
> > >>>> > > >> >>
> > >>>> > > >> >> St.Ack
> > >>>> > > >> >>
> > >>>> > > >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net>
> > >>>> wrote:
> > >>>> > > >> >>
> > >>>> > > >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <stack@duboce.net
> >
> > >>>> wrote:
> > >>>> > > >> >>>
> > >>>> > > >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <
> > >>>> > busbey@cloudera.com>
> > >>>> > > >> >>>> wrote:
> > >>>> > > >> >>>>
> > >>>> > > >> >>>>> I don't recall anything, but I'll go dig through the
> > infra@
> > >>>> > list
> > >>>> > > >> >>>>> later this
> > >>>> > > >> >>>>> evening to make sure.
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>>> Is this on all the H* build hosts?
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>> Seemingly, yes.
> > >>>> > > >> >>>>
> > >>>> > > >> >>>> I can look at infra changes, no prob. Sean. Was just
> > >>>> wondering if
> > >>>> > > >> >>>> anyone knew of a change.
> > >>>> > > >> >>>>
> > >>>> > > >> >>>>
> > >>>> > > >> >>> I don't see anything in INFRA JIRA mailing list related.
> > >>>> > > >> >>>
> > >>>> > > >> >>> Let me file an INFRA BUILDS issue and see if anything
> comes
> > >>>> of it.
> > >>>> > > >> >>>
> > >>>> > > >> >>> St.Ack
> > >>>> > > >> >>>
> > >>>> > > >> >>>
> > >>>> > > >> >>>
> > >>>> > > >> >>>> Thanks,
> > >>>> > > >> >>>> St.Ack
> > >>>> > > >> >>>>
> > >>>> > > >> >>>>
> > >>>> > > >> >>>>
> > >>>> > > >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <
> stack@duboce.net
> > >
> > >>>> > wrote:
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>>> > Trunk builds are failing with OOME, can't create
> native
> > >>>> thread
> > >>>> > > >> >>>>> > (HBASE-14262)
> > >>>> > > >> >>>>> >
> > >>>> > > >> >>>>> > Anyone know of any recent INFRA changes?
> > >>>> > > >> >>>>> >
> > >>>> > > >> >>>>> > Imperfect triage has our problem starting in build
> > >>>> #15134, Aug
> > >>>> > > 18,
> > >>>> > > >> >>>>> 2015
> > >>>> > > >> >>>>> > 5:10:02 AM
> > >>>> > > >> >>>>> >
> > >>>> > > >> >>>>> > Otherwise, was going to mess with heap sizings and
> > >>>> trying to
> > >>>> > > make
> > >>>> > > >> >>>>> the big
> > >>>> > > >> >>>>> > tests smaller so we can get trunk passing again.
> > >>>> > > >> >>>>> >
> > >>>> > > >> >>>>> > Thanks,
> > >>>> > > >> >>>>> > St.Ack
> > >>>> > > >> >>>>> >
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>>> --
> > >>>> > > >> >>>>> Sean
> > >>>> > > >> >>>>>
> > >>>> > > >> >>>>
> > >>>> > > >> >>>>
> > >>>> > > >> >>>
> > >>>> > > >> >>
> > >>>> > > >> >
> > >>>> > > >>
> > >>>> > > >
> > >>>> > > >
> > >>>> > > >
> > >>>> > > > --
> > >>>> > > > Sean
> > >>>> > > >
> > >>>> > >
> > >>>> > >
> > >>>> > >
> > >>>> > > --
> > >>>> > > Sean
> > >>>> > >
> > >>>> >
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Sean
> > >>>>
> > >>>
> > >>>
> > >>
> > >
> >
>
>
>
> --
> Sean
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Sean Busbey <bu...@cloudera.com>.
I updated the job to export the location of Maven 3.0.5 and then hard-set
MAVEN_HOME to point at that version. For now, I've un-blacklisted H0, H4,
H5, and H6 since we have emails saying they were removed for maven. I think
H2 was the javax-inject dependency but I can't find a note about it or H10
or H11 at the moment.
(I also fixed a bug where we wouldn't fail the build if test-patch failed
but we were then able to properly list the problematic javax-inject
dependency files in our local repo.)
On Tue, Sep 15, 2015 at 7:31 AM, Stack <st...@duboce.net> wrote:
> Filed https://issues.apache.org/jira/browse/INFRA-10429 on missing mvn.
> St.Ack
>
> On Mon, Sep 14, 2015 at 7:42 PM, Stack <st...@duboce.net> wrote:
>
> > Removed H4. It's forgotten where mvn is.
> >
> > Will try to talk to infra about getting this fixed on !H0 && !H10 && !H2
> > && !H6 && !H5 && !H4
> >
> > St.Ack
> >
> > On Thu, Sep 10, 2015 at 3:09 PM, Stack <st...@duboce.net> wrote:
> >
> >> I removed H5 and H6 from rotation. They can't find mvn.
> >> St.Ack
> >>
> >> On Fri, Sep 4, 2015 at 10:42 AM, Stack <st...@duboce.net> wrote:
> >>
> >>> You the man Sean!
> >>>
> >>> For those following along, this fail looks like this in the logs:
> >>>
> >>> "Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList
> >>> at META-INF/LICENSE.vm[line 1619, column 22]:
> InvocationTargetException:
> >>> Index: 0, Size: 0 -> [Help 1]"
> >>> St.Ack
> >>>
> >>> On Fri, Sep 4, 2015 at 10:24 AM, Sean Busbey <bu...@cloudera.com>
> >>> wrote:
> >>>
> >>>> Recent run found that javax.inject is the dependency that is somehow
> >>>> getting no license locally on some build boxes. it looks fine on
> maven.
> >>>>
> >>>> I've added steps to the precommit build that ensures this particular
> >>>> dependency is pulled down each time and shows us its contents in the
> >>>> event
> >>>> that things fail.
> >>>>
> >>>> On Mon, Aug 31, 2015 at 11:17 AM, Stack <st...@duboce.net> wrote:
> >>>>
> >>>> > Thanks Sean. That seems to get us going again. I pushed Anoops
> >>>> HBASE-13158
> >>>> > and its running.
> >>>> > St.Ack
> >>>> >
> >>>> > On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <bu...@cloudera.com>
> >>>> wrote:
> >>>> >
> >>>> > > and now I've removed H0 from the precommit rotation, because of
> >>>> repeated
> >>>> > "I
> >>>> > > can't find maven" errors. :/
> >>>> > >
> >>>> > > On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <
> busbey@cloudera.com>
> >>>> > wrote:
> >>>> > >
> >>>> > > > FYI, I've changed the precommit test to run using the
> >>>> test-patch.sh
> >>>> > file
> >>>> > > > from a branch for HBASE-14337 so I can make changes needed to
> >>>> allow the
> >>>> > > > workspace to have a maven repo just for our builds (rather than
> >>>> the
> >>>> > > shared
> >>>> > > > one with the jenkins user).
> >>>> > > >
> >>>> > > > submitted patches are still applied and run against their branch
> >>>> > (default
> >>>> > > > master).
> >>>> > > >
> >>>> > > > Once I have a coherent patch I'll post it to HBASE-14337 and we
> >>>> can
> >>>> > stop
> >>>> > > > using the branch.
> >>>> > > >
> >>>> > > > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net>
> wrote:
> >>>> > > >
> >>>> > > >> A deadlock was causing TestDistributedLogSplitting to hang.
> >>>> Elliott
> >>>> > > fixed
> >>>> > > >> it over in HBASE-14274. This was the source of the zombie
> >>>> processes
> >>>> > > >> reported by INFRA (at least, it matched the one instance they
> >>>> > reported).
> >>>> > > >>
> >>>> > > >> Patch builds were disabled overnight (for a different reason)
> >>>> but are
> >>>> > > back
> >>>> > > >> again.
> >>>> > > >>
> >>>> > > >> St.Ack
> >>>> > > >>
> >>>> > > >>
> >>>> > > >>
> >>>> > > >>
> >>>> > > >>
> >>>> > > >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net>
> wrote:
> >>>> > > >>
> >>>> > > >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net>
> >>>> wrote:
> >>>> > > >> >
> >>>> > > >> >> https://issues.apache.org/jira/browse/BUILDS-109
> >>>> > > >> >>
> >>>> > > >> >
> >>>> > > >> > This became
> https://issues.apache.org/jira/browse/INFRA-10150
> >>>> which
> >>>> > > >> > becomes BUILDS-111.
> >>>> > > >> >
> >>>> > > >> > Errant left-over java processes sucking up resources. Trunk
> is
> >>>> > > >> > particularly susceptible because we spin up way more threads
> >>>> than in
> >>>> > > >> other
> >>>> > > >> > branches (async rpc client). See HBASE-14262 for follow-up.
> >>>> > > >> >
> >>>> > > >> > St.Ack
> >>>> > > >> >
> >>>> > > >> >
> >>>> > > >> >>
> >>>> > > >> >> St.Ack
> >>>> > > >> >>
> >>>> > > >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net>
> >>>> wrote:
> >>>> > > >> >>
> >>>> > > >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net>
> >>>> wrote:
> >>>> > > >> >>>
> >>>> > > >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <
> >>>> > busbey@cloudera.com>
> >>>> > > >> >>>> wrote:
> >>>> > > >> >>>>
> >>>> > > >> >>>>> I don't recall anything, but I'll go dig through the
> infra@
> >>>> > list
> >>>> > > >> >>>>> later this
> >>>> > > >> >>>>> evening to make sure.
> >>>> > > >> >>>>>
> >>>> > > >> >>>>> Is this on all the H* build hosts?
> >>>> > > >> >>>>>
> >>>> > > >> >>>>>
> >>>> > > >> >>>> Seemingly, yes.
> >>>> > > >> >>>>
> >>>> > > >> >>>> I can look at infra changes, no prob. Sean. Was just
> >>>> wondering if
> >>>> > > >> >>>> anyone knew of a change.
> >>>> > > >> >>>>
> >>>> > > >> >>>>
> >>>> > > >> >>> I don't see anything in INFRA JIRA mailing list related.
> >>>> > > >> >>>
> >>>> > > >> >>> Let me file an INFRA BUILDS issue and see if anything comes
> >>>> of it.
> >>>> > > >> >>>
> >>>> > > >> >>> St.Ack
> >>>> > > >> >>>
> >>>> > > >> >>>
> >>>> > > >> >>>
> >>>> > > >> >>>> Thanks,
> >>>> > > >> >>>> St.Ack
> >>>> > > >> >>>>
> >>>> > > >> >>>>
> >>>> > > >> >>>>
> >>>> > > >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <stack@duboce.net
> >
> >>>> > wrote:
> >>>> > > >> >>>>>
> >>>> > > >> >>>>> > Trunk builds are failing with OOME, can't create native
> >>>> thread
> >>>> > > >> >>>>> > (HBASE-14262)
> >>>> > > >> >>>>> >
> >>>> > > >> >>>>> > Anyone know of any recent INFRA changes?
> >>>> > > >> >>>>> >
> >>>> > > >> >>>>> > Imperfect triage has our problem starting in build
> >>>> #15134, Aug
> >>>> > > 18,
> >>>> > > >> >>>>> 2015
> >>>> > > >> >>>>> > 5:10:02 AM
> >>>> > > >> >>>>> >
> >>>> > > >> >>>>> > Otherwise, was going to mess with heap sizings and
> >>>> trying to
> >>>> > > make
> >>>> > > >> >>>>> the big
> >>>> > > >> >>>>> > tests smaller so we can get trunk passing again.
> >>>> > > >> >>>>> >
> >>>> > > >> >>>>> > Thanks,
> >>>> > > >> >>>>> > St.Ack
> >>>> > > >> >>>>> >
> >>>> > > >> >>>>>
> >>>> > > >> >>>>>
> >>>> > > >> >>>>>
> >>>> > > >> >>>>> --
> >>>> > > >> >>>>> Sean
> >>>> > > >> >>>>>
> >>>> > > >> >>>>
> >>>> > > >> >>>>
> >>>> > > >> >>>
> >>>> > > >> >>
> >>>> > > >> >
> >>>> > > >>
> >>>> > > >
> >>>> > > >
> >>>> > > >
> >>>> > > > --
> >>>> > > > Sean
> >>>> > > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > --
> >>>> > > Sean
> >>>> > >
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sean
> >>>>
> >>>
> >>>
> >>
> >
>
--
Sean
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
Filed https://issues.apache.org/jira/browse/INFRA-10429 on missing mvn.
St.Ack
On Mon, Sep 14, 2015 at 7:42 PM, Stack <st...@duboce.net> wrote:
> Removed H4. It's forgotten where mvn is.
>
> Will try to talk to infra about getting this fixed on !H0 && !H10 && !H2
> && !H6 && !H5 && !H4
>
> St.Ack
>
> On Thu, Sep 10, 2015 at 3:09 PM, Stack <st...@duboce.net> wrote:
>
>> I removed H5 and H6 from rotation. They can't find mvn.
>> St.Ack
>>
>> On Fri, Sep 4, 2015 at 10:42 AM, Stack <st...@duboce.net> wrote:
>>
>>> You the man Sean!
>>>
>>> For those following along, this fail looks like this in the logs:
>>>
>>> "Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList
>>> at META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException:
>>> Index: 0, Size: 0 -> [Help 1]"
>>> St.Ack
>>>
>>> On Fri, Sep 4, 2015 at 10:24 AM, Sean Busbey <bu...@cloudera.com>
>>> wrote:
>>>
>>>> Recent run found that javax.inject is the dependency that is somehow
>>>> getting no license locally on some build boxes. it looks fine on maven.
>>>>
>>>> I've added steps to the precommit build that ensures this particular
>>>> dependency is pulled down each time and shows us its contents in the
>>>> event
>>>> that things fail.
>>>>
>>>> On Mon, Aug 31, 2015 at 11:17 AM, Stack <st...@duboce.net> wrote:
>>>>
>>>> > Thanks Sean. That seems to get us going again. I pushed Anoops
>>>> HBASE-13158
>>>> > and its running.
>>>> > St.Ack
>>>> >
>>>> > On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <bu...@cloudera.com>
>>>> wrote:
>>>> >
>>>> > > and now I've removed H0 from the precommit rotation, because of
>>>> repeated
>>>> > "I
>>>> > > can't find maven" errors. :/
>>>> > >
>>>> > > On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <bu...@cloudera.com>
>>>> > wrote:
>>>> > >
>>>> > > > FYI, I've changed the precommit test to run using the
>>>> test-patch.sh
>>>> > file
>>>> > > > from a branch for HBASE-14337 so I can make changes needed to
>>>> allow the
>>>> > > > workspace to have a maven repo just for our builds (rather than
>>>> the
>>>> > > shared
>>>> > > > one with the jenkins user).
>>>> > > >
>>>> > > > submitted patches are still applied and run against their branch
>>>> > (default
>>>> > > > master).
>>>> > > >
>>>> > > > Once I have a coherent patch I'll post it to HBASE-14337 and we
>>>> can
>>>> > stop
>>>> > > > using the branch.
>>>> > > >
>>>> > > > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
>>>> > > >
>>>> > > >> A deadlock was causing TestDistributedLogSplitting to hang.
>>>> Elliott
>>>> > > fixed
>>>> > > >> it over in HBASE-14274. This was the source of the zombie
>>>> processes
>>>> > > >> reported by INFRA (at least, it matched the one instance they
>>>> > reported).
>>>> > > >>
>>>> > > >> Patch builds were disabled overnight (for a different reason)
>>>> but are
>>>> > > back
>>>> > > >> again.
>>>> > > >>
>>>> > > >> St.Ack
>>>> > > >>
>>>> > > >>
>>>> > > >>
>>>> > > >>
>>>> > > >>
>>>> > > >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
>>>> > > >>
>>>> > > >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>> > > >> >
>>>> > > >> >> https://issues.apache.org/jira/browse/BUILDS-109
>>>> > > >> >>
>>>> > > >> >
>>>> > > >> > This became https://issues.apache.org/jira/browse/INFRA-10150
>>>> which
>>>> > > >> > becomes BUILDS-111.
>>>> > > >> >
>>>> > > >> > Errant left-over java processes sucking up resources. Trunk is
>>>> > > >> > particularly susceptible because we spin up way more threads
>>>> than in
>>>> > > >> other
>>>> > > >> > branches (async rpc client). See HBASE-14262 for follow-up.
>>>> > > >> >
>>>> > > >> > St.Ack
>>>> > > >> >
>>>> > > >> >
>>>> > > >> >>
>>>> > > >> >> St.Ack
>>>> > > >> >>
>>>> > > >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>> > > >> >>
>>>> > > >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net>
>>>> wrote:
>>>> > > >> >>>
>>>> > > >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <
>>>> > busbey@cloudera.com>
>>>> > > >> >>>> wrote:
>>>> > > >> >>>>
>>>> > > >> >>>>> I don't recall anything, but I'll go dig through the infra@
>>>> > list
>>>> > > >> >>>>> later this
>>>> > > >> >>>>> evening to make sure.
>>>> > > >> >>>>>
>>>> > > >> >>>>> Is this on all the H* build hosts?
>>>> > > >> >>>>>
>>>> > > >> >>>>>
>>>> > > >> >>>> Seemingly, yes.
>>>> > > >> >>>>
>>>> > > >> >>>> I can look at infra changes, no prob. Sean. Was just
>>>> wondering if
>>>> > > >> >>>> anyone knew of a change.
>>>> > > >> >>>>
>>>> > > >> >>>>
>>>> > > >> >>> I don't see anything in INFRA JIRA mailing list related.
>>>> > > >> >>>
>>>> > > >> >>> Let me file an INFRA BUILDS issue and see if anything comes
>>>> of it.
>>>> > > >> >>>
>>>> > > >> >>> St.Ack
>>>> > > >> >>>
>>>> > > >> >>>
>>>> > > >> >>>
>>>> > > >> >>>> Thanks,
>>>> > > >> >>>> St.Ack
>>>> > > >> >>>>
>>>> > > >> >>>>
>>>> > > >> >>>>
>>>> > > >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net>
>>>> > wrote:
>>>> > > >> >>>>>
>>>> > > >> >>>>> > Trunk builds are failing with OOME, can't create native
>>>> thread
>>>> > > >> >>>>> > (HBASE-14262)
>>>> > > >> >>>>> >
>>>> > > >> >>>>> > Anyone know of any recent INFRA changes?
>>>> > > >> >>>>> >
>>>> > > >> >>>>> > Imperfect triage has our problem starting in build
>>>> #15134, Aug
>>>> > > 18,
>>>> > > >> >>>>> 2015
>>>> > > >> >>>>> > 5:10:02 AM
>>>> > > >> >>>>> >
>>>> > > >> >>>>> > Otherwise, was going to mess with heap sizings and
>>>> trying to
>>>> > > make
>>>> > > >> >>>>> the big
>>>> > > >> >>>>> > tests smaller so we can get trunk passing again.
>>>> > > >> >>>>> >
>>>> > > >> >>>>> > Thanks,
>>>> > > >> >>>>> > St.Ack
>>>> > > >> >>>>> >
>>>> > > >> >>>>>
>>>> > > >> >>>>>
>>>> > > >> >>>>>
>>>> > > >> >>>>> --
>>>> > > >> >>>>> Sean
>>>> > > >> >>>>>
>>>> > > >> >>>>
>>>> > > >> >>>>
>>>> > > >> >>>
>>>> > > >> >>
>>>> > > >> >
>>>> > > >>
>>>> > > >
>>>> > > >
>>>> > > >
>>>> > > > --
>>>> > > > Sean
>>>> > > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Sean
>>>> > >
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Sean
>>>>
>>>
>>>
>>
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
Removed H4. It's forgotten where mvn is.
Will try to talk to infra about getting this fixed on !H0 && !H10 && !H2 &&
!H6 && !H5 && !H4
St.Ack
On Thu, Sep 10, 2015 at 3:09 PM, Stack <st...@duboce.net> wrote:
> I removed H5 and H6 from rotation. They can't find mvn.
> St.Ack
>
> On Fri, Sep 4, 2015 at 10:42 AM, Stack <st...@duboce.net> wrote:
>
>> You the man Sean!
>>
>> For those following along, this fail looks like this in the logs:
>>
>> "Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList at
>> META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException:
>> Index: 0, Size: 0 -> [Help 1]"
>> St.Ack
>>
>> On Fri, Sep 4, 2015 at 10:24 AM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> Recent run found that javax.inject is the dependency that is somehow
>>> getting no license locally on some build boxes. it looks fine on maven.
>>>
>>> I've added steps to the precommit build that ensures this particular
>>> dependency is pulled down each time and shows us its contents in the
>>> event
>>> that things fail.
>>>
>>> On Mon, Aug 31, 2015 at 11:17 AM, Stack <st...@duboce.net> wrote:
>>>
>>> > Thanks Sean. That seems to get us going again. I pushed Anoops
>>> HBASE-13158
>>> > and its running.
>>> > St.Ack
>>> >
>>> > On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <bu...@cloudera.com>
>>> wrote:
>>> >
>>> > > and now I've removed H0 from the precommit rotation, because of
>>> repeated
>>> > "I
>>> > > can't find maven" errors. :/
>>> > >
>>> > > On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <bu...@cloudera.com>
>>> > wrote:
>>> > >
>>> > > > FYI, I've changed the precommit test to run using the test-patch.sh
>>> > file
>>> > > > from a branch for HBASE-14337 so I can make changes needed to
>>> allow the
>>> > > > workspace to have a maven repo just for our builds (rather than the
>>> > > shared
>>> > > > one with the jenkins user).
>>> > > >
>>> > > > submitted patches are still applied and run against their branch
>>> > (default
>>> > > > master).
>>> > > >
>>> > > > Once I have a coherent patch I'll post it to HBASE-14337 and we can
>>> > stop
>>> > > > using the branch.
>>> > > >
>>> > > > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
>>> > > >
>>> > > >> A deadlock was causing TestDistributedLogSplitting to hang.
>>> Elliott
>>> > > fixed
>>> > > >> it over in HBASE-14274. This was the source of the zombie
>>> processes
>>> > > >> reported by INFRA (at least, it matched the one instance they
>>> > reported).
>>> > > >>
>>> > > >> Patch builds were disabled overnight (for a different reason) but
>>> are
>>> > > back
>>> > > >> again.
>>> > > >>
>>> > > >> St.Ack
>>> > > >>
>>> > > >>
>>> > > >>
>>> > > >>
>>> > > >>
>>> > > >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
>>> > > >>
>>> > > >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net>
>>> wrote:
>>> > > >> >
>>> > > >> >> https://issues.apache.org/jira/browse/BUILDS-109
>>> > > >> >>
>>> > > >> >
>>> > > >> > This became https://issues.apache.org/jira/browse/INFRA-10150
>>> which
>>> > > >> > becomes BUILDS-111.
>>> > > >> >
>>> > > >> > Errant left-over java processes sucking up resources. Trunk is
>>> > > >> > particularly susceptible because we spin up way more threads
>>> than in
>>> > > >> other
>>> > > >> > branches (async rpc client). See HBASE-14262 for follow-up.
>>> > > >> >
>>> > > >> > St.Ack
>>> > > >> >
>>> > > >> >
>>> > > >> >>
>>> > > >> >> St.Ack
>>> > > >> >>
>>> > > >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net>
>>> wrote:
>>> > > >> >>
>>> > > >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net>
>>> wrote:
>>> > > >> >>>
>>> > > >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <
>>> > busbey@cloudera.com>
>>> > > >> >>>> wrote:
>>> > > >> >>>>
>>> > > >> >>>>> I don't recall anything, but I'll go dig through the infra@
>>> > list
>>> > > >> >>>>> later this
>>> > > >> >>>>> evening to make sure.
>>> > > >> >>>>>
>>> > > >> >>>>> Is this on all the H* build hosts?
>>> > > >> >>>>>
>>> > > >> >>>>>
>>> > > >> >>>> Seemingly, yes.
>>> > > >> >>>>
>>> > > >> >>>> I can look at infra changes, no prob. Sean. Was just
>>> wondering if
>>> > > >> >>>> anyone knew of a change.
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>> I don't see anything in INFRA JIRA mailing list related.
>>> > > >> >>>
>>> > > >> >>> Let me file an INFRA BUILDS issue and see if anything comes
>>> of it.
>>> > > >> >>>
>>> > > >> >>> St.Ack
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>
>>> > > >> >>>> Thanks,
>>> > > >> >>>> St.Ack
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net>
>>> > wrote:
>>> > > >> >>>>>
>>> > > >> >>>>> > Trunk builds are failing with OOME, can't create native
>>> thread
>>> > > >> >>>>> > (HBASE-14262)
>>> > > >> >>>>> >
>>> > > >> >>>>> > Anyone know of any recent INFRA changes?
>>> > > >> >>>>> >
>>> > > >> >>>>> > Imperfect triage has our problem starting in build
>>> #15134, Aug
>>> > > 18,
>>> > > >> >>>>> 2015
>>> > > >> >>>>> > 5:10:02 AM
>>> > > >> >>>>> >
>>> > > >> >>>>> > Otherwise, was going to mess with heap sizings and trying
>>> to
>>> > > make
>>> > > >> >>>>> the big
>>> > > >> >>>>> > tests smaller so we can get trunk passing again.
>>> > > >> >>>>> >
>>> > > >> >>>>> > Thanks,
>>> > > >> >>>>> > St.Ack
>>> > > >> >>>>> >
>>> > > >> >>>>>
>>> > > >> >>>>>
>>> > > >> >>>>>
>>> > > >> >>>>> --
>>> > > >> >>>>> Sean
>>> > > >> >>>>>
>>> > > >> >>>>
>>> > > >> >>>>
>>> > > >> >>>
>>> > > >> >>
>>> > > >> >
>>> > > >>
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Sean
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Sean
>>> > >
>>> >
>>>
>>>
>>>
>>> --
>>> Sean
>>>
>>
>>
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
I removed H5 and H6 from rotation. They can't find mvn.
St.Ack
On Fri, Sep 4, 2015 at 10:42 AM, Stack <st...@duboce.net> wrote:
> You the man Sean!
>
> For those following along, this fail looks like this in the logs:
>
> "Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException:
> Index: 0, Size: 0 -> [Help 1]"
> St.Ack
>
> On Fri, Sep 4, 2015 at 10:24 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> Recent run found that javax.inject is the dependency that is somehow
>> getting no license locally on some build boxes. it looks fine on maven.
>>
>> I've added steps to the precommit build that ensures this particular
>> dependency is pulled down each time and shows us its contents in the event
>> that things fail.
>>
>> On Mon, Aug 31, 2015 at 11:17 AM, Stack <st...@duboce.net> wrote:
>>
>> > Thanks Sean. That seems to get us going again. I pushed Anoops
>> HBASE-13158
>> > and its running.
>> > St.Ack
>> >
>> > On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <bu...@cloudera.com>
>> wrote:
>> >
>> > > and now I've removed H0 from the precommit rotation, because of
>> repeated
>> > "I
>> > > can't find maven" errors. :/
>> > >
>> > > On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <bu...@cloudera.com>
>> > wrote:
>> > >
>> > > > FYI, I've changed the precommit test to run using the test-patch.sh
>> > file
>> > > > from a branch for HBASE-14337 so I can make changes needed to allow
>> the
>> > > > workspace to have a maven repo just for our builds (rather than the
>> > > shared
>> > > > one with the jenkins user).
>> > > >
>> > > > submitted patches are still applied and run against their branch
>> > (default
>> > > > master).
>> > > >
>> > > > Once I have a coherent patch I'll post it to HBASE-14337 and we can
>> > stop
>> > > > using the branch.
>> > > >
>> > > > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
>> > > >
>> > > >> A deadlock was causing TestDistributedLogSplitting to hang. Elliott
>> > > fixed
>> > > >> it over in HBASE-14274. This was the source of the zombie processes
>> > > >> reported by INFRA (at least, it matched the one instance they
>> > reported).
>> > > >>
>> > > >> Patch builds were disabled overnight (for a different reason) but
>> are
>> > > back
>> > > >> again.
>> > > >>
>> > > >> St.Ack
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
>> > > >>
>> > > >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
>> > > >> >
>> > > >> >> https://issues.apache.org/jira/browse/BUILDS-109
>> > > >> >>
>> > > >> >
>> > > >> > This became https://issues.apache.org/jira/browse/INFRA-10150
>> which
>> > > >> > becomes BUILDS-111.
>> > > >> >
>> > > >> > Errant left-over java processes sucking up resources. Trunk is
>> > > >> > particularly susceptible because we spin up way more threads
>> than in
>> > > >> other
>> > > >> > branches (async rpc client). See HBASE-14262 for follow-up.
>> > > >> >
>> > > >> > St.Ack
>> > > >> >
>> > > >> >
>> > > >> >>
>> > > >> >> St.Ack
>> > > >> >>
>> > > >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net>
>> wrote:
>> > > >> >>
>> > > >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net>
>> wrote:
>> > > >> >>>
>> > > >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <
>> > busbey@cloudera.com>
>> > > >> >>>> wrote:
>> > > >> >>>>
>> > > >> >>>>> I don't recall anything, but I'll go dig through the infra@
>> > list
>> > > >> >>>>> later this
>> > > >> >>>>> evening to make sure.
>> > > >> >>>>>
>> > > >> >>>>> Is this on all the H* build hosts?
>> > > >> >>>>>
>> > > >> >>>>>
>> > > >> >>>> Seemingly, yes.
>> > > >> >>>>
>> > > >> >>>> I can look at infra changes, no prob. Sean. Was just
>> wondering if
>> > > >> >>>> anyone knew of a change.
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>> I don't see anything in INFRA JIRA mailing list related.
>> > > >> >>>
>> > > >> >>> Let me file an INFRA BUILDS issue and see if anything comes of
>> it.
>> > > >> >>>
>> > > >> >>> St.Ack
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>
>> > > >> >>>> Thanks,
>> > > >> >>>> St.Ack
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net>
>> > wrote:
>> > > >> >>>>>
>> > > >> >>>>> > Trunk builds are failing with OOME, can't create native
>> thread
>> > > >> >>>>> > (HBASE-14262)
>> > > >> >>>>> >
>> > > >> >>>>> > Anyone know of any recent INFRA changes?
>> > > >> >>>>> >
>> > > >> >>>>> > Imperfect triage has our problem starting in build #15134,
>> Aug
>> > > 18,
>> > > >> >>>>> 2015
>> > > >> >>>>> > 5:10:02 AM
>> > > >> >>>>> >
>> > > >> >>>>> > Otherwise, was going to mess with heap sizings and trying
>> to
>> > > make
>> > > >> >>>>> the big
>> > > >> >>>>> > tests smaller so we can get trunk passing again.
>> > > >> >>>>> >
>> > > >> >>>>> > Thanks,
>> > > >> >>>>> > St.Ack
>> > > >> >>>>> >
>> > > >> >>>>>
>> > > >> >>>>>
>> > > >> >>>>>
>> > > >> >>>>> --
>> > > >> >>>>> Sean
>> > > >> >>>>>
>> > > >> >>>>
>> > > >> >>>>
>> > > >> >>>
>> > > >> >>
>> > > >> >
>> > > >>
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Sean
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Sean
>> > >
>> >
>>
>>
>>
>> --
>> Sean
>>
>
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
You the man Sean!
For those following along, this fail looks like this in the logs:
"Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList at
META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException:
Index: 0, Size: 0 -> [Help 1]"
St.Ack
On Fri, Sep 4, 2015 at 10:24 AM, Sean Busbey <bu...@cloudera.com> wrote:
> Recent run found that javax.inject is the dependency that is somehow
> getting no license locally on some build boxes. it looks fine on maven.
>
> I've added steps to the precommit build that ensures this particular
> dependency is pulled down each time and shows us its contents in the event
> that things fail.
>
> On Mon, Aug 31, 2015 at 11:17 AM, Stack <st...@duboce.net> wrote:
>
> > Thanks Sean. That seems to get us going again. I pushed Anoops
> HBASE-13158
> > and its running.
> > St.Ack
> >
> > On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > and now I've removed H0 from the precommit rotation, because of
> repeated
> > "I
> > > can't find maven" errors. :/
> > >
> > > On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <bu...@cloudera.com>
> > wrote:
> > >
> > > > FYI, I've changed the precommit test to run using the test-patch.sh
> > file
> > > > from a branch for HBASE-14337 so I can make changes needed to allow
> the
> > > > workspace to have a maven repo just for our builds (rather than the
> > > shared
> > > > one with the jenkins user).
> > > >
> > > > submitted patches are still applied and run against their branch
> > (default
> > > > master).
> > > >
> > > > Once I have a coherent patch I'll post it to HBASE-14337 and we can
> > stop
> > > > using the branch.
> > > >
> > > > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
> > > >
> > > >> A deadlock was causing TestDistributedLogSplitting to hang. Elliott
> > > fixed
> > > >> it over in HBASE-14274. This was the source of the zombie processes
> > > >> reported by INFRA (at least, it matched the one instance they
> > reported).
> > > >>
> > > >> Patch builds were disabled overnight (for a different reason) but
> are
> > > back
> > > >> again.
> > > >>
> > > >> St.Ack
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
> > > >>
> > > >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
> > > >> >
> > > >> >> https://issues.apache.org/jira/browse/BUILDS-109
> > > >> >>
> > > >> >
> > > >> > This became https://issues.apache.org/jira/browse/INFRA-10150
> which
> > > >> > becomes BUILDS-111.
> > > >> >
> > > >> > Errant left-over java processes sucking up resources. Trunk is
> > > >> > particularly susceptible because we spin up way more threads than
> in
> > > >> other
> > > >> > branches (async rpc client). See HBASE-14262 for follow-up.
> > > >> >
> > > >> > St.Ack
> > > >> >
> > > >> >
> > > >> >>
> > > >> >> St.Ack
> > > >> >>
> > > >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
> > > >> >>
> > > >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net>
> wrote:
> > > >> >>>
> > > >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <
> > busbey@cloudera.com>
> > > >> >>>> wrote:
> > > >> >>>>
> > > >> >>>>> I don't recall anything, but I'll go dig through the infra@
> > list
> > > >> >>>>> later this
> > > >> >>>>> evening to make sure.
> > > >> >>>>>
> > > >> >>>>> Is this on all the H* build hosts?
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>> Seemingly, yes.
> > > >> >>>>
> > > >> >>>> I can look at infra changes, no prob. Sean. Was just wondering
> if
> > > >> >>>> anyone knew of a change.
> > > >> >>>>
> > > >> >>>>
> > > >> >>> I don't see anything in INFRA JIRA mailing list related.
> > > >> >>>
> > > >> >>> Let me file an INFRA BUILDS issue and see if anything comes of
> it.
> > > >> >>>
> > > >> >>> St.Ack
> > > >> >>>
> > > >> >>>
> > > >> >>>
> > > >> >>>> Thanks,
> > > >> >>>> St.Ack
> > > >> >>>>
> > > >> >>>>
> > > >> >>>>
> > > >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net>
> > wrote:
> > > >> >>>>>
> > > >> >>>>> > Trunk builds are failing with OOME, can't create native
> thread
> > > >> >>>>> > (HBASE-14262)
> > > >> >>>>> >
> > > >> >>>>> > Anyone know of any recent INFRA changes?
> > > >> >>>>> >
> > > >> >>>>> > Imperfect triage has our problem starting in build #15134,
> Aug
> > > 18,
> > > >> >>>>> 2015
> > > >> >>>>> > 5:10:02 AM
> > > >> >>>>> >
> > > >> >>>>> > Otherwise, was going to mess with heap sizings and trying to
> > > make
> > > >> >>>>> the big
> > > >> >>>>> > tests smaller so we can get trunk passing again.
> > > >> >>>>> >
> > > >> >>>>> > Thanks,
> > > >> >>>>> > St.Ack
> > > >> >>>>> >
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>>
> > > >> >>>>> --
> > > >> >>>>> Sean
> > > >> >>>>>
> > > >> >>>>
> > > >> >>>>
> > > >> >>>
> > > >> >>
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Sean
> > > >
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
>
>
>
> --
> Sean
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Sean Busbey <bu...@cloudera.com>.
Recent run found that javax.inject is the dependency that is somehow
getting no license locally on some build boxes. it looks fine on maven.
I've added steps to the precommit build that ensures this particular
dependency is pulled down each time and shows us its contents in the event
that things fail.
On Mon, Aug 31, 2015 at 11:17 AM, Stack <st...@duboce.net> wrote:
> Thanks Sean. That seems to get us going again. I pushed Anoops HBASE-13158
> and its running.
> St.Ack
>
> On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > and now I've removed H0 from the precommit rotation, because of repeated
> "I
> > can't find maven" errors. :/
> >
> > On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <bu...@cloudera.com>
> wrote:
> >
> > > FYI, I've changed the precommit test to run using the test-patch.sh
> file
> > > from a branch for HBASE-14337 so I can make changes needed to allow the
> > > workspace to have a maven repo just for our builds (rather than the
> > shared
> > > one with the jenkins user).
> > >
> > > submitted patches are still applied and run against their branch
> (default
> > > master).
> > >
> > > Once I have a coherent patch I'll post it to HBASE-14337 and we can
> stop
> > > using the branch.
> > >
> > > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
> > >
> > >> A deadlock was causing TestDistributedLogSplitting to hang. Elliott
> > fixed
> > >> it over in HBASE-14274. This was the source of the zombie processes
> > >> reported by INFRA (at least, it matched the one instance they
> reported).
> > >>
> > >> Patch builds were disabled overnight (for a different reason) but are
> > back
> > >> again.
> > >>
> > >> St.Ack
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
> > >>
> > >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
> > >> >
> > >> >> https://issues.apache.org/jira/browse/BUILDS-109
> > >> >>
> > >> >
> > >> > This became https://issues.apache.org/jira/browse/INFRA-10150 which
> > >> > becomes BUILDS-111.
> > >> >
> > >> > Errant left-over java processes sucking up resources. Trunk is
> > >> > particularly susceptible because we spin up way more threads than in
> > >> other
> > >> > branches (async rpc client). See HBASE-14262 for follow-up.
> > >> >
> > >> > St.Ack
> > >> >
> > >> >
> > >> >>
> > >> >> St.Ack
> > >> >>
> > >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
> > >> >>
> > >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
> > >> >>>
> > >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <
> busbey@cloudera.com>
> > >> >>>> wrote:
> > >> >>>>
> > >> >>>>> I don't recall anything, but I'll go dig through the infra@
> list
> > >> >>>>> later this
> > >> >>>>> evening to make sure.
> > >> >>>>>
> > >> >>>>> Is this on all the H* build hosts?
> > >> >>>>>
> > >> >>>>>
> > >> >>>> Seemingly, yes.
> > >> >>>>
> > >> >>>> I can look at infra changes, no prob. Sean. Was just wondering if
> > >> >>>> anyone knew of a change.
> > >> >>>>
> > >> >>>>
> > >> >>> I don't see anything in INFRA JIRA mailing list related.
> > >> >>>
> > >> >>> Let me file an INFRA BUILDS issue and see if anything comes of it.
> > >> >>>
> > >> >>> St.Ack
> > >> >>>
> > >> >>>
> > >> >>>
> > >> >>>> Thanks,
> > >> >>>> St.Ack
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net>
> wrote:
> > >> >>>>>
> > >> >>>>> > Trunk builds are failing with OOME, can't create native thread
> > >> >>>>> > (HBASE-14262)
> > >> >>>>> >
> > >> >>>>> > Anyone know of any recent INFRA changes?
> > >> >>>>> >
> > >> >>>>> > Imperfect triage has our problem starting in build #15134, Aug
> > 18,
> > >> >>>>> 2015
> > >> >>>>> > 5:10:02 AM
> > >> >>>>> >
> > >> >>>>> > Otherwise, was going to mess with heap sizings and trying to
> > make
> > >> >>>>> the big
> > >> >>>>> > tests smaller so we can get trunk passing again.
> > >> >>>>> >
> > >> >>>>> > Thanks,
> > >> >>>>> > St.Ack
> > >> >>>>> >
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> --
> > >> >>>>> Sean
> > >> >>>>>
> > >> >>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Sean
> > >
> >
> >
> >
> > --
> > Sean
> >
>
--
Sean
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
Thanks Sean. That seems to get us going again. I pushed Anoops HBASE-13158
and its running.
St.Ack
On Mon, Aug 31, 2015 at 8:54 AM, Sean Busbey <bu...@cloudera.com> wrote:
> and now I've removed H0 from the precommit rotation, because of repeated "I
> can't find maven" errors. :/
>
> On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <bu...@cloudera.com> wrote:
>
> > FYI, I've changed the precommit test to run using the test-patch.sh file
> > from a branch for HBASE-14337 so I can make changes needed to allow the
> > workspace to have a maven repo just for our builds (rather than the
> shared
> > one with the jenkins user).
> >
> > submitted patches are still applied and run against their branch (default
> > master).
> >
> > Once I have a coherent patch I'll post it to HBASE-14337 and we can stop
> > using the branch.
> >
> > On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
> >
> >> A deadlock was causing TestDistributedLogSplitting to hang. Elliott
> fixed
> >> it over in HBASE-14274. This was the source of the zombie processes
> >> reported by INFRA (at least, it matched the one instance they reported).
> >>
> >> Patch builds were disabled overnight (for a different reason) but are
> back
> >> again.
> >>
> >> St.Ack
> >>
> >>
> >>
> >>
> >>
> >> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
> >>
> >> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
> >> >
> >> >> https://issues.apache.org/jira/browse/BUILDS-109
> >> >>
> >> >
> >> > This became https://issues.apache.org/jira/browse/INFRA-10150 which
> >> > becomes BUILDS-111.
> >> >
> >> > Errant left-over java processes sucking up resources. Trunk is
> >> > particularly susceptible because we spin up way more threads than in
> >> other
> >> > branches (async rpc client). See HBASE-14262 for follow-up.
> >> >
> >> > St.Ack
> >> >
> >> >
> >> >>
> >> >> St.Ack
> >> >>
> >> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
> >> >>
> >> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
> >> >>>
> >> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>>> I don't recall anything, but I'll go dig through the infra@ list
> >> >>>>> later this
> >> >>>>> evening to make sure.
> >> >>>>>
> >> >>>>> Is this on all the H* build hosts?
> >> >>>>>
> >> >>>>>
> >> >>>> Seemingly, yes.
> >> >>>>
> >> >>>> I can look at infra changes, no prob. Sean. Was just wondering if
> >> >>>> anyone knew of a change.
> >> >>>>
> >> >>>>
> >> >>> I don't see anything in INFRA JIRA mailing list related.
> >> >>>
> >> >>> Let me file an INFRA BUILDS issue and see if anything comes of it.
> >> >>>
> >> >>> St.Ack
> >> >>>
> >> >>>
> >> >>>
> >> >>>> Thanks,
> >> >>>> St.Ack
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
> >> >>>>>
> >> >>>>> > Trunk builds are failing with OOME, can't create native thread
> >> >>>>> > (HBASE-14262)
> >> >>>>> >
> >> >>>>> > Anyone know of any recent INFRA changes?
> >> >>>>> >
> >> >>>>> > Imperfect triage has our problem starting in build #15134, Aug
> 18,
> >> >>>>> 2015
> >> >>>>> > 5:10:02 AM
> >> >>>>> >
> >> >>>>> > Otherwise, was going to mess with heap sizings and trying to
> make
> >> >>>>> the big
> >> >>>>> > tests smaller so we can get trunk passing again.
> >> >>>>> >
> >> >>>>> > Thanks,
> >> >>>>> > St.Ack
> >> >>>>> >
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> --
> >> >>>>> Sean
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>
> >> >>
> >> >
> >>
> >
> >
> >
> > --
> > Sean
> >
>
>
>
> --
> Sean
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Sean Busbey <bu...@cloudera.com>.
and now I've removed H0 from the precommit rotation, because of repeated "I
can't find maven" errors. :/
On Mon, Aug 31, 2015 at 10:01 AM, Sean Busbey <bu...@cloudera.com> wrote:
> FYI, I've changed the precommit test to run using the test-patch.sh file
> from a branch for HBASE-14337 so I can make changes needed to allow the
> workspace to have a maven repo just for our builds (rather than the shared
> one with the jenkins user).
>
> submitted patches are still applied and run against their branch (default
> master).
>
> Once I have a coherent patch I'll post it to HBASE-14337 and we can stop
> using the branch.
>
> On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
>
>> A deadlock was causing TestDistributedLogSplitting to hang. Elliott fixed
>> it over in HBASE-14274. This was the source of the zombie processes
>> reported by INFRA (at least, it matched the one instance they reported).
>>
>> Patch builds were disabled overnight (for a different reason) but are back
>> again.
>>
>> St.Ack
>>
>>
>>
>>
>>
>> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
>>
>> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
>> >
>> >> https://issues.apache.org/jira/browse/BUILDS-109
>> >>
>> >
>> > This became https://issues.apache.org/jira/browse/INFRA-10150 which
>> > becomes BUILDS-111.
>> >
>> > Errant left-over java processes sucking up resources. Trunk is
>> > particularly susceptible because we spin up way more threads than in
>> other
>> > branches (async rpc client). See HBASE-14262 for follow-up.
>> >
>> > St.Ack
>> >
>> >
>> >>
>> >> St.Ack
>> >>
>> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
>> >>
>> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
>> >>>
>> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com>
>> >>>> wrote:
>> >>>>
>> >>>>> I don't recall anything, but I'll go dig through the infra@ list
>> >>>>> later this
>> >>>>> evening to make sure.
>> >>>>>
>> >>>>> Is this on all the H* build hosts?
>> >>>>>
>> >>>>>
>> >>>> Seemingly, yes.
>> >>>>
>> >>>> I can look at infra changes, no prob. Sean. Was just wondering if
>> >>>> anyone knew of a change.
>> >>>>
>> >>>>
>> >>> I don't see anything in INFRA JIRA mailing list related.
>> >>>
>> >>> Let me file an INFRA BUILDS issue and see if anything comes of it.
>> >>>
>> >>> St.Ack
>> >>>
>> >>>
>> >>>
>> >>>> Thanks,
>> >>>> St.Ack
>> >>>>
>> >>>>
>> >>>>
>> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
>> >>>>>
>> >>>>> > Trunk builds are failing with OOME, can't create native thread
>> >>>>> > (HBASE-14262)
>> >>>>> >
>> >>>>> > Anyone know of any recent INFRA changes?
>> >>>>> >
>> >>>>> > Imperfect triage has our problem starting in build #15134, Aug 18,
>> >>>>> 2015
>> >>>>> > 5:10:02 AM
>> >>>>> >
>> >>>>> > Otherwise, was going to mess with heap sizings and trying to make
>> >>>>> the big
>> >>>>> > tests smaller so we can get trunk passing again.
>> >>>>> >
>> >>>>> > Thanks,
>> >>>>> > St.Ack
>> >>>>> >
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> --
>> >>>>> Sean
>> >>>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
>
>
> --
> Sean
>
--
Sean
Re: Testing and CI -- Apache Jenkins Builds
Posted by Sean Busbey <bu...@cloudera.com>.
FYI, I've changed the precommit test to run using the test-patch.sh file
from a branch for HBASE-14337 so I can make changes needed to allow the
workspace to have a maven repo just for our builds (rather than the shared
one with the jenkins user).
submitted patches are still applied and run against their branch (default
master).
Once I have a coherent patch I'll post it to HBASE-14337 and we can stop
using the branch.
On Fri, Aug 21, 2015 at 2:17 PM, Stack <st...@duboce.net> wrote:
> A deadlock was causing TestDistributedLogSplitting to hang. Elliott fixed
> it over in HBASE-14274. This was the source of the zombie processes
> reported by INFRA (at least, it matched the one instance they reported).
>
> Patch builds were disabled overnight (for a different reason) but are back
> again.
>
> St.Ack
>
>
>
>
>
> On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
>
> > On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
> >
> >> https://issues.apache.org/jira/browse/BUILDS-109
> >>
> >
> > This became https://issues.apache.org/jira/browse/INFRA-10150 which
> > becomes BUILDS-111.
> >
> > Errant left-over java processes sucking up resources. Trunk is
> > particularly susceptible because we spin up way more threads than in
> other
> > branches (async rpc client). See HBASE-14262 for follow-up.
> >
> > St.Ack
> >
> >
> >>
> >> St.Ack
> >>
> >> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
> >>
> >>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
> >>>
> >>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> I don't recall anything, but I'll go dig through the infra@ list
> >>>>> later this
> >>>>> evening to make sure.
> >>>>>
> >>>>> Is this on all the H* build hosts?
> >>>>>
> >>>>>
> >>>> Seemingly, yes.
> >>>>
> >>>> I can look at infra changes, no prob. Sean. Was just wondering if
> >>>> anyone knew of a change.
> >>>>
> >>>>
> >>> I don't see anything in INFRA JIRA mailing list related.
> >>>
> >>> Let me file an INFRA BUILDS issue and see if anything comes of it.
> >>>
> >>> St.Ack
> >>>
> >>>
> >>>
> >>>> Thanks,
> >>>> St.Ack
> >>>>
> >>>>
> >>>>
> >>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
> >>>>>
> >>>>> > Trunk builds are failing with OOME, can't create native thread
> >>>>> > (HBASE-14262)
> >>>>> >
> >>>>> > Anyone know of any recent INFRA changes?
> >>>>> >
> >>>>> > Imperfect triage has our problem starting in build #15134, Aug 18,
> >>>>> 2015
> >>>>> > 5:10:02 AM
> >>>>> >
> >>>>> > Otherwise, was going to mess with heap sizings and trying to make
> >>>>> the big
> >>>>> > tests smaller so we can get trunk passing again.
> >>>>> >
> >>>>> > Thanks,
> >>>>> > St.Ack
> >>>>> >
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Sean
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
--
Sean
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
A deadlock was causing TestDistributedLogSplitting to hang. Elliott fixed
it over in HBASE-14274. This was the source of the zombie processes
reported by INFRA (at least, it matched the one instance they reported).
Patch builds were disabled overnight (for a different reason) but are back
again.
St.Ack
On Thu, Aug 20, 2015 at 9:44 AM, Stack <st...@duboce.net> wrote:
> On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
>
>> https://issues.apache.org/jira/browse/BUILDS-109
>>
>
> This became https://issues.apache.org/jira/browse/INFRA-10150 which
> becomes BUILDS-111.
>
> Errant left-over java processes sucking up resources. Trunk is
> particularly susceptible because we spin up way more threads than in other
> branches (async rpc client). See HBASE-14262 for follow-up.
>
> St.Ack
>
>
>>
>> St.Ack
>>
>> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
>>
>>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
>>>
>>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I don't recall anything, but I'll go dig through the infra@ list
>>>>> later this
>>>>> evening to make sure.
>>>>>
>>>>> Is this on all the H* build hosts?
>>>>>
>>>>>
>>>> Seemingly, yes.
>>>>
>>>> I can look at infra changes, no prob. Sean. Was just wondering if
>>>> anyone knew of a change.
>>>>
>>>>
>>> I don't see anything in INFRA JIRA mailing list related.
>>>
>>> Let me file an INFRA BUILDS issue and see if anything comes of it.
>>>
>>> St.Ack
>>>
>>>
>>>
>>>> Thanks,
>>>> St.Ack
>>>>
>>>>
>>>>
>>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
>>>>>
>>>>> > Trunk builds are failing with OOME, can't create native thread
>>>>> > (HBASE-14262)
>>>>> >
>>>>> > Anyone know of any recent INFRA changes?
>>>>> >
>>>>> > Imperfect triage has our problem starting in build #15134, Aug 18,
>>>>> 2015
>>>>> > 5:10:02 AM
>>>>> >
>>>>> > Otherwise, was going to mess with heap sizings and trying to make
>>>>> the big
>>>>> > tests smaller so we can get trunk passing again.
>>>>> >
>>>>> > Thanks,
>>>>> > St.Ack
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sean
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
On Wed, Aug 19, 2015 at 6:18 PM, Stack <st...@duboce.net> wrote:
> https://issues.apache.org/jira/browse/BUILDS-109
>
This became https://issues.apache.org/jira/browse/INFRA-10150 which becomes
BUILDS-111.
Errant left-over java processes sucking up resources. Trunk is particularly
susceptible because we spin up way more threads than in other branches
(async rpc client). See HBASE-14262 for follow-up.
St.Ack
>
> St.Ack
>
> On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
>
>> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
>>
>>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com>
>>> wrote:
>>>
>>>> I don't recall anything, but I'll go dig through the infra@ list later
>>>> this
>>>> evening to make sure.
>>>>
>>>> Is this on all the H* build hosts?
>>>>
>>>>
>>> Seemingly, yes.
>>>
>>> I can look at infra changes, no prob. Sean. Was just wondering if anyone
>>> knew of a change.
>>>
>>>
>> I don't see anything in INFRA JIRA mailing list related.
>>
>> Let me file an INFRA BUILDS issue and see if anything comes of it.
>>
>> St.Ack
>>
>>
>>
>>> Thanks,
>>> St.Ack
>>>
>>>
>>>
>>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
>>>>
>>>> > Trunk builds are failing with OOME, can't create native thread
>>>> > (HBASE-14262)
>>>> >
>>>> > Anyone know of any recent INFRA changes?
>>>> >
>>>> > Imperfect triage has our problem starting in build #15134, Aug 18,
>>>> 2015
>>>> > 5:10:02 AM
>>>> >
>>>> > Otherwise, was going to mess with heap sizings and trying to make the
>>>> big
>>>> > tests smaller so we can get trunk passing again.
>>>> >
>>>> > Thanks,
>>>> > St.Ack
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Sean
>>>>
>>>
>>>
>>
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
https://issues.apache.org/jira/browse/BUILDS-109
St.Ack
On Wed, Aug 19, 2015 at 5:38 PM, Stack <st...@duboce.net> wrote:
> On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
>
>> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com> wrote:
>>
>>> I don't recall anything, but I'll go dig through the infra@ list later
>>> this
>>> evening to make sure.
>>>
>>> Is this on all the H* build hosts?
>>>
>>>
>> Seemingly, yes.
>>
>> I can look at infra changes, no prob. Sean. Was just wondering if anyone
>> knew of a change.
>>
>>
> I don't see anything in INFRA JIRA mailing list related.
>
> Let me file an INFRA BUILDS issue and see if anything comes of it.
>
> St.Ack
>
>
>
>> Thanks,
>> St.Ack
>>
>>
>>
>>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
>>>
>>> > Trunk builds are failing with OOME, can't create native thread
>>> > (HBASE-14262)
>>> >
>>> > Anyone know of any recent INFRA changes?
>>> >
>>> > Imperfect triage has our problem starting in build #15134, Aug 18, 2015
>>> > 5:10:02 AM
>>> >
>>> > Otherwise, was going to mess with heap sizings and trying to make the
>>> big
>>> > tests smaller so we can get trunk passing again.
>>> >
>>> > Thanks,
>>> > St.Ack
>>> >
>>>
>>>
>>>
>>> --
>>> Sean
>>>
>>
>>
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
On Wed, Aug 19, 2015 at 5:24 PM, Stack <st...@duboce.net> wrote:
> On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com> wrote:
>
>> I don't recall anything, but I'll go dig through the infra@ list later
>> this
>> evening to make sure.
>>
>> Is this on all the H* build hosts?
>>
>>
> Seemingly, yes.
>
> I can look at infra changes, no prob. Sean. Was just wondering if anyone
> knew of a change.
>
>
I don't see anything in INFRA JIRA mailing list related.
Let me file an INFRA BUILDS issue and see if anything comes of it.
St.Ack
> Thanks,
> St.Ack
>
>
>
>> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
>>
>> > Trunk builds are failing with OOME, can't create native thread
>> > (HBASE-14262)
>> >
>> > Anyone know of any recent INFRA changes?
>> >
>> > Imperfect triage has our problem starting in build #15134, Aug 18, 2015
>> > 5:10:02 AM
>> >
>> > Otherwise, was going to mess with heap sizings and trying to make the
>> big
>> > tests smaller so we can get trunk passing again.
>> >
>> > Thanks,
>> > St.Ack
>> >
>>
>>
>>
>> --
>> Sean
>>
>
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Stack <st...@duboce.net>.
On Wed, Aug 19, 2015 at 5:08 PM, Sean Busbey <bu...@cloudera.com> wrote:
> I don't recall anything, but I'll go dig through the infra@ list later
> this
> evening to make sure.
>
> Is this on all the H* build hosts?
>
>
Seemingly, yes.
I can look at infra changes, no prob. Sean. Was just wondering if anyone
knew of a change.
Thanks,
St.Ack
> On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
>
> > Trunk builds are failing with OOME, can't create native thread
> > (HBASE-14262)
> >
> > Anyone know of any recent INFRA changes?
> >
> > Imperfect triage has our problem starting in build #15134, Aug 18, 2015
> > 5:10:02 AM
> >
> > Otherwise, was going to mess with heap sizings and trying to make the big
> > tests smaller so we can get trunk passing again.
> >
> > Thanks,
> > St.Ack
> >
>
>
>
> --
> Sean
>
Re: Testing and CI -- Apache Jenkins Builds
Posted by Sean Busbey <bu...@cloudera.com>.
I don't recall anything, but I'll go dig through the infra@ list later this
evening to make sure.
Is this on all the H* build hosts?
On Wed, Aug 19, 2015 at 7:05 PM, Stack <st...@duboce.net> wrote:
> Trunk builds are failing with OOME, can't create native thread
> (HBASE-14262)
>
> Anyone know of any recent INFRA changes?
>
> Imperfect triage has our problem starting in build #15134, Aug 18, 2015
> 5:10:02 AM
>
> Otherwise, was going to mess with heap sizings and trying to make the big
> tests smaller so we can get trunk passing again.
>
> Thanks,
> St.Ack
>
--
Sean