You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andrew Purtell <ap...@apache.org> on 2014/06/25 03:09:33 UTC
Planning to roll the 0.98.4 RC on 6/30
Planning to roll the 0.98.4 RC on Monday 6/30.
I should have done it this week. Sorry, got a little sidetracked.
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
Single unit test runs don't indicate anything Ted, by definition of
intermittent.
On Thu, Jun 26, 2014 at 3:40 PM, Ted Yu <yu...@gmail.com> wrote:
> I ran 0.98 test suite using hadoop-2 and it passed.
>
> Here is the OS:
> Linux k.net 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13
> 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> I will run the suite again.
>
> FYI
>
> [INFO] HBase ............................................. SUCCESS [1.577s]
> [INFO] HBase - Common .................................... SUCCESS
> [20.219s]
> [INFO] HBase - Protocol .................................. SUCCESS [0.286s]
> [INFO] HBase - Client .................................... SUCCESS
> [36.419s]
> [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [5.076s]
> [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [1.506s]
> [INFO] HBase - Prefix Tree ............................... SUCCESS [2.605s]
> [INFO] HBase - Server .................................... SUCCESS
> [1:08:43.945s]
> [INFO] HBase - Testing Util .............................. SUCCESS [1.242s]
> [INFO] HBase - Thrift .................................... SUCCESS
> [2:02.400s]
> [INFO] HBase - Shell ..................................... SUCCESS
> [1:41.594s]
> [INFO] HBase - Integration Tests ......................... SUCCESS [0.895s]
> [INFO] HBase - Examples .................................. SUCCESS [5.421s]
> [INFO] HBase - Assembly .................................. SUCCESS [0.895s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1:13:44.733s
> [INFO] Finished at: Thu Jun 26 22:02:13 UTC 2014
>
>
> On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > I'm finding that repeated runs of the unit test suite at the head of
> branch
> > 0.98 intermittently fail. Individual tests do not, so this likely a
> lagging
> > shutdown, port/resource conflict, and/or zombie test issue. I am
> currently
> > bisecting commits on 0.98 branch since the last release in the hope of
> > pinning this down to a single change. Depending on how quickly that can
> > happen, the RC might happen on Monday or not. As things stand at the head
> > of the branch, I'd not +1 the RC given the release criteria I've been
> using
> > up to now.
> >
> >
> > On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Planning to roll the 0.98.4 RC on Monday 6/30.
> > >
> > > I should have done it this week. Sorry, got a little sidetracked.
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Ted Yu <yu...@gmail.com>.
I didn't set any custom JVM setting.
Cheers
On Thu, Jun 26, 2014 at 3:43 PM, Mikhail Antonov <ol...@gmail.com>
wrote:
> Ted,
>
> do you have customized JVM settings for -Xmx, -XX:MaxPermGen?
>
> -Mikhail
>
> 2014-06-26 15:40 GMT-07:00, Ted Yu <yu...@gmail.com>:
> > I ran 0.98 test suite using hadoop-2 and it passed.
> >
> > Here is the OS:
> > Linux k.net 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13
> > 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux
> >
> > I will run the suite again.
> >
> > FYI
> >
> > [INFO] HBase ............................................. SUCCESS
> [1.577s]
> > [INFO] HBase - Common .................................... SUCCESS
> > [20.219s]
> > [INFO] HBase - Protocol .................................. SUCCESS
> [0.286s]
> > [INFO] HBase - Client .................................... SUCCESS
> > [36.419s]
> > [INFO] HBase - Hadoop Compatibility ...................... SUCCESS
> [5.076s]
> > [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS
> [1.506s]
> > [INFO] HBase - Prefix Tree ............................... SUCCESS
> [2.605s]
> > [INFO] HBase - Server .................................... SUCCESS
> > [1:08:43.945s]
> > [INFO] HBase - Testing Util .............................. SUCCESS
> [1.242s]
> > [INFO] HBase - Thrift .................................... SUCCESS
> > [2:02.400s]
> > [INFO] HBase - Shell ..................................... SUCCESS
> > [1:41.594s]
> > [INFO] HBase - Integration Tests ......................... SUCCESS
> [0.895s]
> > [INFO] HBase - Examples .................................. SUCCESS
> [5.421s]
> > [INFO] HBase - Assembly .................................. SUCCESS
> [0.895s]
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] BUILD SUCCESS
> > [INFO]
> > ------------------------------------------------------------------------
> > [INFO] Total time: 1:13:44.733s
> > [INFO] Finished at: Thu Jun 26 22:02:13 UTC 2014
> >
> >
> > On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >> I'm finding that repeated runs of the unit test suite at the head of
> >> branch
> >> 0.98 intermittently fail. Individual tests do not, so this likely a
> >> lagging
> >> shutdown, port/resource conflict, and/or zombie test issue. I am
> >> currently
> >> bisecting commits on 0.98 branch since the last release in the hope of
> >> pinning this down to a single change. Depending on how quickly that can
> >> happen, the RC might happen on Monday or not. As things stand at the
> head
> >> of the branch, I'd not +1 the RC given the release criteria I've been
> >> using
> >> up to now.
> >>
> >>
> >> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>
> >> > Planning to roll the 0.98.4 RC on Monday 6/30.
> >> >
> >> > I should have done it this week. Sorry, got a little sidetracked.
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> > - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet
> >> > Hein
> >> > (via Tom White)
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >> - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >
>
>
> --
> Thanks,
> Michael Antonov
>
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Mikhail Antonov <ol...@gmail.com>.
Ted,
do you have customized JVM settings for -Xmx, -XX:MaxPermGen?
-Mikhail
2014-06-26 15:40 GMT-07:00, Ted Yu <yu...@gmail.com>:
> I ran 0.98 test suite using hadoop-2 and it passed.
>
> Here is the OS:
> Linux k.net 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13
> 11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux
>
> I will run the suite again.
>
> FYI
>
> [INFO] HBase ............................................. SUCCESS [1.577s]
> [INFO] HBase - Common .................................... SUCCESS
> [20.219s]
> [INFO] HBase - Protocol .................................. SUCCESS [0.286s]
> [INFO] HBase - Client .................................... SUCCESS
> [36.419s]
> [INFO] HBase - Hadoop Compatibility ...................... SUCCESS [5.076s]
> [INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [1.506s]
> [INFO] HBase - Prefix Tree ............................... SUCCESS [2.605s]
> [INFO] HBase - Server .................................... SUCCESS
> [1:08:43.945s]
> [INFO] HBase - Testing Util .............................. SUCCESS [1.242s]
> [INFO] HBase - Thrift .................................... SUCCESS
> [2:02.400s]
> [INFO] HBase - Shell ..................................... SUCCESS
> [1:41.594s]
> [INFO] HBase - Integration Tests ......................... SUCCESS [0.895s]
> [INFO] HBase - Examples .................................. SUCCESS [5.421s]
> [INFO] HBase - Assembly .................................. SUCCESS [0.895s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 1:13:44.733s
> [INFO] Finished at: Thu Jun 26 22:02:13 UTC 2014
>
>
> On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
>> I'm finding that repeated runs of the unit test suite at the head of
>> branch
>> 0.98 intermittently fail. Individual tests do not, so this likely a
>> lagging
>> shutdown, port/resource conflict, and/or zombie test issue. I am
>> currently
>> bisecting commits on 0.98 branch since the last release in the hope of
>> pinning this down to a single change. Depending on how quickly that can
>> happen, the RC might happen on Monday or not. As things stand at the head
>> of the branch, I'd not +1 the RC given the release criteria I've been
>> using
>> up to now.
>>
>>
>> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>> > Planning to roll the 0.98.4 RC on Monday 6/30.
>> >
>> > I should have done it this week. Sorry, got a little sidetracked.
>> >
>> > --
>> > Best regards,
>> >
>> > - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet
>> > Hein
>> > (via Tom White)
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>
--
Thanks,
Michael Antonov
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Ted Yu <yu...@gmail.com>.
I ran 0.98 test suite using hadoop-2 and it passed.
Here is the OS:
Linux k.net 2.6.32-220.23.1.el6.YAHOO.20120713.x86_64 #1 SMP Fri Jul 13
11:40:51 CDT 2012 x86_64 x86_64 x86_64 GNU/Linux
I will run the suite again.
FYI
[INFO] HBase ............................................. SUCCESS [1.577s]
[INFO] HBase - Common .................................... SUCCESS [20.219s]
[INFO] HBase - Protocol .................................. SUCCESS [0.286s]
[INFO] HBase - Client .................................... SUCCESS [36.419s]
[INFO] HBase - Hadoop Compatibility ...................... SUCCESS [5.076s]
[INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [1.506s]
[INFO] HBase - Prefix Tree ............................... SUCCESS [2.605s]
[INFO] HBase - Server .................................... SUCCESS
[1:08:43.945s]
[INFO] HBase - Testing Util .............................. SUCCESS [1.242s]
[INFO] HBase - Thrift .................................... SUCCESS
[2:02.400s]
[INFO] HBase - Shell ..................................... SUCCESS
[1:41.594s]
[INFO] HBase - Integration Tests ......................... SUCCESS [0.895s]
[INFO] HBase - Examples .................................. SUCCESS [5.421s]
[INFO] HBase - Assembly .................................. SUCCESS [0.895s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 1:13:44.733s
[INFO] Finished at: Thu Jun 26 22:02:13 UTC 2014
On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org> wrote:
> I'm finding that repeated runs of the unit test suite at the head of branch
> 0.98 intermittently fail. Individual tests do not, so this likely a lagging
> shutdown, port/resource conflict, and/or zombie test issue. I am currently
> bisecting commits on 0.98 branch since the last release in the hope of
> pinning this down to a single change. Depending on how quickly that can
> happen, the RC might happen on Monday or not. As things stand at the head
> of the branch, I'd not +1 the RC given the release criteria I've been using
> up to now.
>
>
> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Planning to roll the 0.98.4 RC on Monday 6/30.
> >
> > I should have done it this week. Sorry, got a little sidetracked.
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Konstantin Boudnik <co...@apache.org>.
Hi Andrew.
Thanks for keep this rolling! I would love to have
HBASE-11422 to get in and will finish the patch tomorrow.
Thanks!
Cos
On Sun, Jun 29, 2014 at 11:35AM, Andrew Purtell wrote:
> A bisect with more test suite repetitions per step lead to 34ae4a9
> (HBASE-11094: Distributed log replay is incompatible for rolling restarts).
> Bisecting again to see if the results are stable.
>
>
> On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org> wrote:
>
> > I'm finding that repeated runs of the unit test suite at the head of
> > branch 0.98 intermittently fail. Individual tests do not, so this likely a
> > lagging shutdown, port/resource conflict, and/or zombie test issue. I am
> > currently bisecting commits on 0.98 branch since the last release in the
> > hope of pinning this down to a single change. Depending on how quickly that
> > can happen, the RC might happen on Monday or not. As things stand at the
> > head of the branch, I'd not +1 the RC given the release criteria I've been
> > using up to now.
> >
> >
> > On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >> Planning to roll the 0.98.4 RC on Monday 6/30.
> >>
> >> I should have done it this week. Sorry, got a little sidetracked.
> >>
> >
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
Around the time I was realizing a stable bisect wasn't going to be likely,
I started receiving alerts from AWS about the testing instance. Shut it
down, started a new one. Not seeing issues any more.
On Mon, Jun 30, 2014 at 12:43 PM, Andrew Purtell <ap...@apache.org>
wrote:
> I did another bisect and landed somewhere else. The results are not
> stable. Trying again with more reps per run. If that doesn't work, we will
> have to treat the symptom.
>
>
> On Sun, Jun 29, 2014 at 11:35 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
>> A bisect with more test suite repetitions per step lead to 34ae4a9
>> (HBASE-11094: Distributed log replay is incompatible for rolling restarts).
>> Bisecting again to see if the results are stable.
>>
>>
>> On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>>> I'm finding that repeated runs of the unit test suite at the head of
>>> branch 0.98 intermittently fail. Individual tests do not, so this likely a
>>> lagging shutdown, port/resource conflict, and/or zombie test issue. I am
>>> currently bisecting commits on 0.98 branch since the last release in the
>>> hope of pinning this down to a single change. Depending on how quickly that
>>> can happen, the RC might happen on Monday or not. As things stand at the
>>> head of the branch, I'd not +1 the RC given the release criteria I've been
>>> using up to now.
>>>
>>>
>>> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
>>> wrote:
>>>
>>>> Planning to roll the 0.98.4 RC on Monday 6/30.
>>>>
>>>> I should have done it this week. Sorry, got a little sidetracked.
>>>>
>>>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
Please go ahead.
On Mon, Jun 30, 2014 at 1:47 PM, Enis Söztutar <en...@apache.org> wrote:
> I want to commit https://issues.apache.org/jira/browse/HBASE-11335 to 0.98
> branch as well. Should I hold on until the analysis, or go ahead?
>
> Enis
>
>
> On Mon, Jun 30, 2014 at 12:43 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > I did another bisect and landed somewhere else. The results are not
> stable.
> > Trying again with more reps per run. If that doesn't work, we will have
> to
> > treat the symptom.
> >
> >
> > On Sun, Jun 29, 2014 at 11:35 AM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > A bisect with more test suite repetitions per step lead to 34ae4a9
> > > (HBASE-11094: Distributed log replay is incompatible for rolling
> > restarts).
> > > Bisecting again to see if the results are stable.
> > >
> > >
> > > On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org>
> > > wrote:
> > >
> > >> I'm finding that repeated runs of the unit test suite at the head of
> > >> branch 0.98 intermittently fail. Individual tests do not, so this
> > likely a
> > >> lagging shutdown, port/resource conflict, and/or zombie test issue. I
> am
> > >> currently bisecting commits on 0.98 branch since the last release in
> the
> > >> hope of pinning this down to a single change. Depending on how quickly
> > that
> > >> can happen, the RC might happen on Monday or not. As things stand at
> the
> > >> head of the branch, I'd not +1 the RC given the release criteria I've
> > been
> > >> using up to now.
> > >>
> > >>
> > >> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> > >> wrote:
> > >>
> > >>> Planning to roll the 0.98.4 RC on Monday 6/30.
> > >>>
> > >>> I should have done it this week. Sorry, got a little sidetracked.
> > >>>
> > >>
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Enis Söztutar <en...@apache.org>.
I want to commit https://issues.apache.org/jira/browse/HBASE-11335 to 0.98
branch as well. Should I hold on until the analysis, or go ahead?
Enis
On Mon, Jun 30, 2014 at 12:43 PM, Andrew Purtell <ap...@apache.org>
wrote:
> I did another bisect and landed somewhere else. The results are not stable.
> Trying again with more reps per run. If that doesn't work, we will have to
> treat the symptom.
>
>
> On Sun, Jun 29, 2014 at 11:35 AM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > A bisect with more test suite repetitions per step lead to 34ae4a9
> > (HBASE-11094: Distributed log replay is incompatible for rolling
> restarts).
> > Bisecting again to see if the results are stable.
> >
> >
> > On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> >> I'm finding that repeated runs of the unit test suite at the head of
> >> branch 0.98 intermittently fail. Individual tests do not, so this
> likely a
> >> lagging shutdown, port/resource conflict, and/or zombie test issue. I am
> >> currently bisecting commits on 0.98 branch since the last release in the
> >> hope of pinning this down to a single change. Depending on how quickly
> that
> >> can happen, the RC might happen on Monday or not. As things stand at the
> >> head of the branch, I'd not +1 the RC given the release criteria I've
> been
> >> using up to now.
> >>
> >>
> >> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> >> wrote:
> >>
> >>> Planning to roll the 0.98.4 RC on Monday 6/30.
> >>>
> >>> I should have done it this week. Sorry, got a little sidetracked.
> >>>
> >>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
I did another bisect and landed somewhere else. The results are not stable.
Trying again with more reps per run. If that doesn't work, we will have to
treat the symptom.
On Sun, Jun 29, 2014 at 11:35 AM, Andrew Purtell <ap...@apache.org>
wrote:
> A bisect with more test suite repetitions per step lead to 34ae4a9
> (HBASE-11094: Distributed log replay is incompatible for rolling restarts).
> Bisecting again to see if the results are stable.
>
>
> On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
>> I'm finding that repeated runs of the unit test suite at the head of
>> branch 0.98 intermittently fail. Individual tests do not, so this likely a
>> lagging shutdown, port/resource conflict, and/or zombie test issue. I am
>> currently bisecting commits on 0.98 branch since the last release in the
>> hope of pinning this down to a single change. Depending on how quickly that
>> can happen, the RC might happen on Monday or not. As things stand at the
>> head of the branch, I'd not +1 the RC given the release criteria I've been
>> using up to now.
>>
>>
>> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
>> wrote:
>>
>>> Planning to roll the 0.98.4 RC on Monday 6/30.
>>>
>>> I should have done it this week. Sorry, got a little sidetracked.
>>>
>>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
A bisect with more test suite repetitions per step lead to 34ae4a9
(HBASE-11094: Distributed log replay is incompatible for rolling restarts).
Bisecting again to see if the results are stable.
On Thu, Jun 26, 2014 at 1:39 PM, Andrew Purtell <ap...@apache.org> wrote:
> I'm finding that repeated runs of the unit test suite at the head of
> branch 0.98 intermittently fail. Individual tests do not, so this likely a
> lagging shutdown, port/resource conflict, and/or zombie test issue. I am
> currently bisecting commits on 0.98 branch since the last release in the
> hope of pinning this down to a single change. Depending on how quickly that
> can happen, the RC might happen on Monday or not. As things stand at the
> head of the branch, I'd not +1 the RC given the release criteria I've been
> using up to now.
>
>
> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
>> Planning to roll the 0.98.4 RC on Monday 6/30.
>>
>> I should have done it this week. Sorry, got a little sidetracked.
>>
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
I haven't tried yet here, but on other occasions I've found the slow or
hanging tests I was after, yeah.
I haven't spent any time testing the master branch with the whole suite
recently.
On Thu, Jun 26, 2014 at 4:59 PM, Mikhail Antonov <ol...@gmail.com>
wrote:
> And if you disable forking completely, do the tests pass for you always, or
> they also fail intermittently?
>
>
> 2014-06-26 15:59 GMT-07:00 Andrew Purtell <ap...@apache.org>:
>
> > Additionally we run unit tests in parallel to reduce the total time
> > required for test suite execution. Surefire will fork multiple JVMs,
> > dynamically generate test jars containing a subset of tests, and run
> them.
> > That can make isolating hanging tests difficult but this behavior can be
> > influenced by defines on the Maven command line. For example, to fork a
> > process for every single unit test:
> >
> > mvn test -Dsurefire.firstPartForkMode=always
> > -Dsurefire.secondPartForkMode=always
> >
> > And then if you find a hanging surefire runner, you can dump thread
> stacks
> > of that JVM and know only the unit test you find methods of in the stacks
> > contributed to the current wedged state.
> >
> >
> > On Thu, Jun 26, 2014 at 3:48 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Java 7u60 64-bit on an EC2 m3.4xlarge. Just running the unit test suite
> > in
> > > a loop. I don't set any special Maven options in MVN_OPTS or anything
> > like
> > > that.
> > >
> > > Historically failures that occur when the suite executes but do not
> when
> > > individual tests pass happen because one test does not shut down in a
> > > timely manner, or at all, and a subsequent test might use the same
> > > hardcoded path or port. When that happens we have a sporadic and
> > sometimes
> > > load sensitive failure. Complicating, each time one clones a repository
> > on
> > > a different host or file filesystem JUnit may pick up a different test
> > > order, influenced by whatever readdir hands back for each package.
> > >
> > >
> > >
> > >
> > > On Thu, Jun 26, 2014 at 3:25 PM, Mikhail Antonov <olorinbant@gmail.com
> >
> > > wrote:
> > >
> > >> Andrew,
> > >>
> > >> Could you share some details - on what env. you're running the tests,
> > and
> > >> at which point do that fail? I'm curious because of lately I'm seeing
> > >> weird
> > >> failures on current master too, which do not happen on hadoop-qa -
> > >> individual tests always pass, but when running the suite tests either
> > get
> > >> stuck and time out (in roughly the same point), or fail with NPE or
> > >> PermGen
> > >> exception. I've been blaming my environment first, but may be it's
> > >> something related.
> > >>
> > >> -Mikhail
> > >>
> > >>
> > >>
> > >>
> > >> 2014-06-26 13:39 GMT-07:00 Andrew Purtell <ap...@apache.org>:
> > >>
> > >> > I'm finding that repeated runs of the unit test suite at the head of
> > >> branch
> > >> > 0.98 intermittently fail. Individual tests do not, so this likely a
> > >> lagging
> > >> > shutdown, port/resource conflict, and/or zombie test issue. I am
> > >> currently
> > >> > bisecting commits on 0.98 branch since the last release in the hope
> of
> > >> > pinning this down to a single change. Depending on how quickly that
> > can
> > >> > happen, the RC might happen on Monday or not. As things stand at the
> > >> head
> > >> > of the branch, I'd not +1 the RC given the release criteria I've
> been
> > >> using
> > >> > up to now.
> > >>
> > >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Mikhail Antonov <ol...@gmail.com>.
And if you disable forking completely, do the tests pass for you always, or
they also fail intermittently?
2014-06-26 15:59 GMT-07:00 Andrew Purtell <ap...@apache.org>:
> Additionally we run unit tests in parallel to reduce the total time
> required for test suite execution. Surefire will fork multiple JVMs,
> dynamically generate test jars containing a subset of tests, and run them.
> That can make isolating hanging tests difficult but this behavior can be
> influenced by defines on the Maven command line. For example, to fork a
> process for every single unit test:
>
> mvn test -Dsurefire.firstPartForkMode=always
> -Dsurefire.secondPartForkMode=always
>
> And then if you find a hanging surefire runner, you can dump thread stacks
> of that JVM and know only the unit test you find methods of in the stacks
> contributed to the current wedged state.
>
>
> On Thu, Jun 26, 2014 at 3:48 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Java 7u60 64-bit on an EC2 m3.4xlarge. Just running the unit test suite
> in
> > a loop. I don't set any special Maven options in MVN_OPTS or anything
> like
> > that.
> >
> > Historically failures that occur when the suite executes but do not when
> > individual tests pass happen because one test does not shut down in a
> > timely manner, or at all, and a subsequent test might use the same
> > hardcoded path or port. When that happens we have a sporadic and
> sometimes
> > load sensitive failure. Complicating, each time one clones a repository
> on
> > a different host or file filesystem JUnit may pick up a different test
> > order, influenced by whatever readdir hands back for each package.
> >
> >
> >
> >
> > On Thu, Jun 26, 2014 at 3:25 PM, Mikhail Antonov <ol...@gmail.com>
> > wrote:
> >
> >> Andrew,
> >>
> >> Could you share some details - on what env. you're running the tests,
> and
> >> at which point do that fail? I'm curious because of lately I'm seeing
> >> weird
> >> failures on current master too, which do not happen on hadoop-qa -
> >> individual tests always pass, but when running the suite tests either
> get
> >> stuck and time out (in roughly the same point), or fail with NPE or
> >> PermGen
> >> exception. I've been blaming my environment first, but may be it's
> >> something related.
> >>
> >> -Mikhail
> >>
> >>
> >>
> >>
> >> 2014-06-26 13:39 GMT-07:00 Andrew Purtell <ap...@apache.org>:
> >>
> >> > I'm finding that repeated runs of the unit test suite at the head of
> >> branch
> >> > 0.98 intermittently fail. Individual tests do not, so this likely a
> >> lagging
> >> > shutdown, port/resource conflict, and/or zombie test issue. I am
> >> currently
> >> > bisecting commits on 0.98 branch since the last release in the hope of
> >> > pinning this down to a single change. Depending on how quickly that
> can
> >> > happen, the RC might happen on Monday or not. As things stand at the
> >> head
> >> > of the branch, I'd not +1 the RC given the release criteria I've been
> >> using
> >> > up to now.
> >>
> >
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Thanks,
Michael Antonov
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
Additionally we run unit tests in parallel to reduce the total time
required for test suite execution. Surefire will fork multiple JVMs,
dynamically generate test jars containing a subset of tests, and run them.
That can make isolating hanging tests difficult but this behavior can be
influenced by defines on the Maven command line. For example, to fork a
process for every single unit test:
mvn test -Dsurefire.firstPartForkMode=always
-Dsurefire.secondPartForkMode=always
And then if you find a hanging surefire runner, you can dump thread stacks
of that JVM and know only the unit test you find methods of in the stacks
contributed to the current wedged state.
On Thu, Jun 26, 2014 at 3:48 PM, Andrew Purtell <ap...@apache.org> wrote:
> Java 7u60 64-bit on an EC2 m3.4xlarge. Just running the unit test suite in
> a loop. I don't set any special Maven options in MVN_OPTS or anything like
> that.
>
> Historically failures that occur when the suite executes but do not when
> individual tests pass happen because one test does not shut down in a
> timely manner, or at all, and a subsequent test might use the same
> hardcoded path or port. When that happens we have a sporadic and sometimes
> load sensitive failure. Complicating, each time one clones a repository on
> a different host or file filesystem JUnit may pick up a different test
> order, influenced by whatever readdir hands back for each package.
>
>
>
>
> On Thu, Jun 26, 2014 at 3:25 PM, Mikhail Antonov <ol...@gmail.com>
> wrote:
>
>> Andrew,
>>
>> Could you share some details - on what env. you're running the tests, and
>> at which point do that fail? I'm curious because of lately I'm seeing
>> weird
>> failures on current master too, which do not happen on hadoop-qa -
>> individual tests always pass, but when running the suite tests either get
>> stuck and time out (in roughly the same point), or fail with NPE or
>> PermGen
>> exception. I've been blaming my environment first, but may be it's
>> something related.
>>
>> -Mikhail
>>
>>
>>
>>
>> 2014-06-26 13:39 GMT-07:00 Andrew Purtell <ap...@apache.org>:
>>
>> > I'm finding that repeated runs of the unit test suite at the head of
>> branch
>> > 0.98 intermittently fail. Individual tests do not, so this likely a
>> lagging
>> > shutdown, port/resource conflict, and/or zombie test issue. I am
>> currently
>> > bisecting commits on 0.98 branch since the last release in the hope of
>> > pinning this down to a single change. Depending on how quickly that can
>> > happen, the RC might happen on Monday or not. As things stand at the
>> head
>> > of the branch, I'd not +1 the RC given the release criteria I've been
>> using
>> > up to now.
>>
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
Java 7u60 64-bit on an EC2 m3.4xlarge. Just running the unit test suite in
a loop. I don't set any special Maven options in MVN_OPTS or anything like
that.
Historically failures that occur when the suite executes but do not when
individual tests pass happen because one test does not shut down in a
timely manner, or at all, and a subsequent test might use the same
hardcoded path or port. When that happens we have a sporadic and sometimes
load sensitive failure. Complicating, each time one clones a repository on
a different host or file filesystem JUnit may pick up a different test
order, influenced by whatever readdir hands back for each package.
On Thu, Jun 26, 2014 at 3:25 PM, Mikhail Antonov <ol...@gmail.com>
wrote:
> Andrew,
>
> Could you share some details - on what env. you're running the tests, and
> at which point do that fail? I'm curious because of lately I'm seeing weird
> failures on current master too, which do not happen on hadoop-qa -
> individual tests always pass, but when running the suite tests either get
> stuck and time out (in roughly the same point), or fail with NPE or PermGen
> exception. I've been blaming my environment first, but may be it's
> something related.
>
> -Mikhail
>
>
>
>
> 2014-06-26 13:39 GMT-07:00 Andrew Purtell <ap...@apache.org>:
>
> > I'm finding that repeated runs of the unit test suite at the head of
> branch
> > 0.98 intermittently fail. Individual tests do not, so this likely a
> lagging
> > shutdown, port/resource conflict, and/or zombie test issue. I am
> currently
> > bisecting commits on 0.98 branch since the last release in the hope of
> > pinning this down to a single change. Depending on how quickly that can
> > happen, the RC might happen on Monday or not. As things stand at the head
> > of the branch, I'd not +1 the RC given the release criteria I've been
> using
> > up to now.
> >
> >
> > On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> >
> > > Planning to roll the 0.98.4 RC on Monday 6/30.
> > >
> > > I should have done it this week. Sorry, got a little sidetracked.
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Mikhail Antonov <ol...@gmail.com>.
Andrew,
Could you share some details - on what env. you're running the tests, and
at which point do that fail? I'm curious because of lately I'm seeing weird
failures on current master too, which do not happen on hadoop-qa -
individual tests always pass, but when running the suite tests either get
stuck and time out (in roughly the same point), or fail with NPE or PermGen
exception. I've been blaming my environment first, but may be it's
something related.
-Mikhail
2014-06-26 13:39 GMT-07:00 Andrew Purtell <ap...@apache.org>:
> I'm finding that repeated runs of the unit test suite at the head of branch
> 0.98 intermittently fail. Individual tests do not, so this likely a lagging
> shutdown, port/resource conflict, and/or zombie test issue. I am currently
> bisecting commits on 0.98 branch since the last release in the hope of
> pinning this down to a single change. Depending on how quickly that can
> happen, the RC might happen on Monday or not. As things stand at the head
> of the branch, I'd not +1 the RC given the release criteria I've been using
> up to now.
>
>
> On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org>
> wrote:
>
> > Planning to roll the 0.98.4 RC on Monday 6/30.
> >
> > I should have done it this week. Sorry, got a little sidetracked.
> >
> > --
> > Best regards,
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Thanks,
Michael Antonov
Re: Planning to roll the 0.98.4 RC on 6/30
Posted by Andrew Purtell <ap...@apache.org>.
I'm finding that repeated runs of the unit test suite at the head of branch
0.98 intermittently fail. Individual tests do not, so this likely a lagging
shutdown, port/resource conflict, and/or zombie test issue. I am currently
bisecting commits on 0.98 branch since the last release in the hope of
pinning this down to a single change. Depending on how quickly that can
happen, the RC might happen on Monday or not. As things stand at the head
of the branch, I'd not +1 the RC given the release criteria I've been using
up to now.
On Tue, Jun 24, 2014 at 6:09 PM, Andrew Purtell <ap...@apache.org> wrote:
> Planning to roll the 0.98.4 RC on Monday 6/30.
>
> I should have done it this week. Sorry, got a little sidetracked.
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)