You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Chetan Mehrotra <ch...@gmail.com> on 2016/11/22 05:11:25 UTC
Frequent failures in standby test
Hi Team,
Since last 4-6 builds I am seeing a recurring failure of few test in
standby module
* FailoverIPRangeIT
* ExternalPrivateStoreIT
* StandbyTestIT
Probably something to be looked into
Chetan Mehrotra
Re: Frequent failures in standby test
Posted by Francesco Mari <ma...@gmail.com>.
In the meantime, I configured those tests not to run when they are
scheduled under a "beam" node. I will send a separate mail to builds@
and put oak-dev@ in CC.
2016-12-08 11:23 GMT+01:00 Robert Munteanu <ro...@apache.org>:
> Hi,
>
> On Thu, 2016-12-08 at 10:01 +0100, Francesco Mari wrote:
>> I analysed the latest builds to figure out what's going on with some
>> tests related to Cold Standby. It seems that the problem is
>> restricted
>> to some Jenkins nodes - see [1] for more details. Can we configure
>> the
>> Jenkins job not to run on nodes labelled "beam"?
>>
>> [1]: https://issues.apache.org/jira/browse/OAK-
>> 5017?focusedCommentId=15731590&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-15731590
>
> The job is configured to only run on 'ubuntu' nodes, but it seems that
> this restriction is not taken into account. The following Jenkins issue
> might be related
>
> https://issues.jenkins-ci.org/browse/JENKINS-23459
>
> Perhaps a question better suited to builds@ ? They would be interested
> in proper resource utilisation.
>
> Robert
>
>>
>> 2016-11-25 13:25 GMT+01:00 Francesco Mari <ma...@gmail.com>:
>> > Every failure in build 1298 is due to "java.net.BindException:
>> > Address
>> > already in use". According to the build log, the port wasn't bound
>> > by
>> > a server started during a test. The first failing test
>> > (ExternalSharedStoreIT) was also the first test in the build to
>> > start
>> > a server. The issue persisted for some tests (the whole range of
>> > tests
>> > in ExternalSharedStoreIT and the first test in FailoverSslTestIT),
>> > so
>> > I can't deduce any random behaviour. Once the port (presumingly
>> > bound
>> > to an external process) was released, every other integration test
>> > worked without any problem.
>> >
>> > 2016-11-25 5:38 GMT+01:00 Chetan Mehrotra <chetan.mehrotra@gmail.co
>> > m>:
>> > > Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20mat
>> > > rix/1298/
>> > > the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8
>> > > looks
>> > > like passed.
>> > > Chetan Mehrotra
>> > >
>> > >
>> > > On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
>> > > <ch...@gmail.com> wrote:
>> > > > They are from oak-segment-tar. See
>> > > > https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matri
>> > > > x/1295/#showFailuresLink
>> > > > Chetan Mehrotra
>> > > >
>> > > >
>> > > > On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
>> > > > <ma...@gmail.com> wrote:
>> > > > > Are those from oak-tarmk-standby or oak-segment-tar?
>> > > > >
>> > > > > 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <chetan.mehrotra@gm
>> > > > > ail.com>:
>> > > > > > Hi Team,
>> > > > > >
>> > > > > > Since last 4-6 builds I am seeing a recurring failure of
>> > > > > > few test in
>> > > > > > standby module
>> > > > > >
>> > > > > > * FailoverIPRangeIT
>> > > > > > * ExternalPrivateStoreIT
>> > > > > > * StandbyTestIT
>> > > > > >
>> > > > > > Probably something to be looked into
>> > > > > >
>> > > > > > Chetan Mehrotra
>
Re: Frequent failures in standby test
Posted by Robert Munteanu <ro...@apache.org>.
Hi,
On Thu, 2016-12-08 at 10:01 +0100, Francesco Mari wrote:
> I analysed the latest builds to figure out what's going on with some
> tests related to Cold Standby. It seems that the problem is
> restricted
> to some Jenkins nodes - see [1] for more details. Can we configure
> the
> Jenkins job not to run on nodes labelled "beam"?
>
> [1]: https://issues.apache.org/jira/browse/OAK-
> 5017?focusedCommentId=15731590&page=com.atlassian.jira.plugin.system.
> issuetabpanels:comment-tabpanel#comment-15731590
The job is configured to only run on 'ubuntu' nodes, but it seems that
this restriction is not taken into account. The following Jenkins issue
might be related
https://issues.jenkins-ci.org/browse/JENKINS-23459
Perhaps a question better suited to builds@ ? They would be interested
in proper resource utilisation.
Robert
>
> 2016-11-25 13:25 GMT+01:00 Francesco Mari <ma...@gmail.com>:
> > Every failure in build 1298 is due to "java.net.BindException:
> > Address
> > already in use". According to the build log, the port wasn't bound
> > by
> > a server started during a test. The first failing test
> > (ExternalSharedStoreIT) was also the first test in the build to
> > start
> > a server. The issue persisted for some tests (the whole range of
> > tests
> > in ExternalSharedStoreIT and the first test in FailoverSslTestIT),
> > so
> > I can't deduce any random behaviour. Once the port (presumingly
> > bound
> > to an external process) was released, every other integration test
> > worked without any problem.
> >
> > 2016-11-25 5:38 GMT+01:00 Chetan Mehrotra <chetan.mehrotra@gmail.co
> > m>:
> > > Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20mat
> > > rix/1298/
> > > the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8
> > > looks
> > > like passed.
> > > Chetan Mehrotra
> > >
> > >
> > > On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
> > > <ch...@gmail.com> wrote:
> > > > They are from oak-segment-tar. See
> > > > https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matri
> > > > x/1295/#showFailuresLink
> > > > Chetan Mehrotra
> > > >
> > > >
> > > > On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
> > > > <ma...@gmail.com> wrote:
> > > > > Are those from oak-tarmk-standby or oak-segment-tar?
> > > > >
> > > > > 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <chetan.mehrotra@gm
> > > > > ail.com>:
> > > > > > Hi Team,
> > > > > >
> > > > > > Since last 4-6 builds I am seeing a recurring failure of
> > > > > > few test in
> > > > > > standby module
> > > > > >
> > > > > > * FailoverIPRangeIT
> > > > > > * ExternalPrivateStoreIT
> > > > > > * StandbyTestIT
> > > > > >
> > > > > > Probably something to be looked into
> > > > > >
> > > > > > Chetan Mehrotra
Re: Frequent failures in standby test
Posted by Francesco Mari <ma...@gmail.com>.
I analysed the latest builds to figure out what's going on with some
tests related to Cold Standby. It seems that the problem is restricted
to some Jenkins nodes - see [1] for more details. Can we configure the
Jenkins job not to run on nodes labelled "beam"?
[1]: https://issues.apache.org/jira/browse/OAK-5017?focusedCommentId=15731590&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15731590
2016-11-25 13:25 GMT+01:00 Francesco Mari <ma...@gmail.com>:
> Every failure in build 1298 is due to "java.net.BindException: Address
> already in use". According to the build log, the port wasn't bound by
> a server started during a test. The first failing test
> (ExternalSharedStoreIT) was also the first test in the build to start
> a server. The issue persisted for some tests (the whole range of tests
> in ExternalSharedStoreIT and the first test in FailoverSslTestIT), so
> I can't deduce any random behaviour. Once the port (presumingly bound
> to an external process) was released, every other integration test
> worked without any problem.
>
> 2016-11-25 5:38 GMT+01:00 Chetan Mehrotra <ch...@gmail.com>:
>> Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1298/
>> the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8 looks
>> like passed.
>> Chetan Mehrotra
>>
>>
>> On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
>> <ch...@gmail.com> wrote:
>>> They are from oak-segment-tar. See
>>> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1295/#showFailuresLink
>>> Chetan Mehrotra
>>>
>>>
>>> On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
>>> <ma...@gmail.com> wrote:
>>>> Are those from oak-tarmk-standby or oak-segment-tar?
>>>>
>>>> 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <ch...@gmail.com>:
>>>>> Hi Team,
>>>>>
>>>>> Since last 4-6 builds I am seeing a recurring failure of few test in
>>>>> standby module
>>>>>
>>>>> * FailoverIPRangeIT
>>>>> * ExternalPrivateStoreIT
>>>>> * StandbyTestIT
>>>>>
>>>>> Probably something to be looked into
>>>>>
>>>>> Chetan Mehrotra
Re: Frequent failures in standby test
Posted by Francesco Mari <ma...@gmail.com>.
Every failure in build 1298 is due to "java.net.BindException: Address
already in use". According to the build log, the port wasn't bound by
a server started during a test. The first failing test
(ExternalSharedStoreIT) was also the first test in the build to start
a server. The issue persisted for some tests (the whole range of tests
in ExternalSharedStoreIT and the first test in FailoverSslTestIT), so
I can't deduce any random behaviour. Once the port (presumingly bound
to an external process) was released, every other integration test
worked without any problem.
2016-11-25 5:38 GMT+01:00 Chetan Mehrotra <ch...@gmail.com>:
> Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1298/
> the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8 looks
> like passed.
> Chetan Mehrotra
>
>
> On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
> <ch...@gmail.com> wrote:
>> They are from oak-segment-tar. See
>> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1295/#showFailuresLink
>> Chetan Mehrotra
>>
>>
>> On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
>> <ma...@gmail.com> wrote:
>>> Are those from oak-tarmk-standby or oak-segment-tar?
>>>
>>> 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <ch...@gmail.com>:
>>>> Hi Team,
>>>>
>>>> Since last 4-6 builds I am seeing a recurring failure of few test in
>>>> standby module
>>>>
>>>> * FailoverIPRangeIT
>>>> * ExternalPrivateStoreIT
>>>> * StandbyTestIT
>>>>
>>>> Probably something to be looked into
>>>>
>>>> Chetan Mehrotra
Re: Frequent failures in standby test
Posted by Chetan Mehrotra <ch...@gmail.com>.
Per https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1298/
the test again failed but mostly on Jdk 1.7. The test on Jdk 1.8 looks
like passed.
Chetan Mehrotra
On Tue, Nov 22, 2016 at 12:48 PM, Chetan Mehrotra
<ch...@gmail.com> wrote:
> They are from oak-segment-tar. See
> https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1295/#showFailuresLink
> Chetan Mehrotra
>
>
> On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
> <ma...@gmail.com> wrote:
>> Are those from oak-tarmk-standby or oak-segment-tar?
>>
>> 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <ch...@gmail.com>:
>>> Hi Team,
>>>
>>> Since last 4-6 builds I am seeing a recurring failure of few test in
>>> standby module
>>>
>>> * FailoverIPRangeIT
>>> * ExternalPrivateStoreIT
>>> * StandbyTestIT
>>>
>>> Probably something to be looked into
>>>
>>> Chetan Mehrotra
Re: Frequent failures in standby test
Posted by Chetan Mehrotra <ch...@gmail.com>.
They are from oak-segment-tar. See
https://builds.apache.org/job/Apache%20Jackrabbit%20Oak%20matrix/1295/#showFailuresLink
Chetan Mehrotra
On Tue, Nov 22, 2016 at 12:42 PM, Francesco Mari
<ma...@gmail.com> wrote:
> Are those from oak-tarmk-standby or oak-segment-tar?
>
> 2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <ch...@gmail.com>:
>> Hi Team,
>>
>> Since last 4-6 builds I am seeing a recurring failure of few test in
>> standby module
>>
>> * FailoverIPRangeIT
>> * ExternalPrivateStoreIT
>> * StandbyTestIT
>>
>> Probably something to be looked into
>>
>> Chetan Mehrotra
Re: Frequent failures in standby test
Posted by Francesco Mari <ma...@gmail.com>.
Are those from oak-tarmk-standby or oak-segment-tar?
2016-11-22 6:11 GMT+01:00 Chetan Mehrotra <ch...@gmail.com>:
> Hi Team,
>
> Since last 4-6 builds I am seeing a recurring failure of few test in
> standby module
>
> * FailoverIPRangeIT
> * ExternalPrivateStoreIT
> * StandbyTestIT
>
> Probably something to be looked into
>
> Chetan Mehrotra