Test run issues on qpid-java trunk?

[Robbie Gemmell <ro...@gmail.com>]

Hi folks,

Irina mentioned seeing some failures for the java build in a CI system
she runs. It has been failing consistently on one box, and
intermittent on another. Taking a quick peek it seems like the root
cause are the "Too many open files" exceptions happening at various
points. One example is failing to open a Selector, another is Logback
failing to create a file for its appender during setUp, and another
was failing to find the class under test. The failures start part way
through the broker-core module.

It isn't clear exactly when the issue started as some of the
intervening reports have already been removed, but it seems like the
last successfull build was from July 2nd, and the first with issues
*may* have been on July 3rd as a result. There looks to have been a
bunch of logging related changes around then and subsequently, so I
thought I'd mention it in case it does signal an issue (it may
alternatively indicate a need for increasing the open files limit).
FWIW, the build passed locally when I tried and seems as happy as
usual on the ASF Jenkins nodes.

Robbie

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Test run issues on qpid-java trunk?

[Robbie Gemmell <ro...@gmail.com>]

Hi Keith,

On one node it has been failing since at least the 7th, but quite
likely since its run on the 3rd. There looks to be 3 commits that
would have been in that run on the 3rd, r1688825, r1688862, and
r1688958, the last of which was for QPID-6613 test log changes.

Another build has run after your revert and just passed on that node,
which is interesting in itself since there didn't seem to be any
change in behaviour resulting from the original commit on the 13th, so
some other changes must be interacting too. In runs before and after
your commit on the 13th the first test to fail was
LocalTransactionTest.testStoreEnqueuesCausesExceptions due to being
unable to load some classes, with the following tests mostly doing the
same except those in TCPandSSLTransportTest (which look to be the last
to run), with its tests instead failing to open a Selector during
SelectorThread init while starting the NetworkConnectionScheduler.

If it happens again I'll let you know. Not sure what details I can get
from the box other than the logs, anything specific you were thinking
of?

Robbie

On 16 July 2015 at 09:03, Keith W <ke...@gmail.com> wrote:
> Hi Robbie
>
> QPID-6613 has been making some changes to test logging.  One area of change
> is the generation of per-test log files which is now handled by Logback's
> SiftingAppender.
>
> I made a commit (r1690741) on Monday (13th) (to help investigate a
> seemingly new sporadical issue where the Java Broker was failing to close
> down completely because the Selector-Port thread remains running).  This
> commit would mean the JVM running the tests would hold on to more file
> handles for longer.  I have reverted this change as I expect that it is
> contributing to the issue you are seeing.  I have just reverted this change.
>
> I wonder if the issue you were observing from around 2nd/3rd July is the
> same as my Selector-Port thread issue.  If you see a recurrence, can you
> post some details?  I am having problems reproducing on the kit I have
> available to me.
>
> cheers, Keith.
>
>
> On 15 July 2015 at 15:24, Robbie Gemmell <ro...@gmail.com> wrote:
>
>> Hi folks,
>>
>> Irina mentioned seeing some failures for the java build in a CI system
>> she runs. It has been failing consistently on one box, and
>> intermittent on another. Taking a quick peek it seems like the root
>> cause are the "Too many open files" exceptions happening at various
>> points. One example is failing to open a Selector, another is Logback
>> failing to create a file for its appender during setUp, and another
>> was failing to find the class under test. The failures start part way
>> through the broker-core module.
>>
>> It isn't clear exactly when the issue started as some of the
>> intervening reports have already been removed, but it seems like the
>> last successfull build was from July 2nd, and the first with issues
>> *may* have been on July 3rd as a result. There looks to have been a
>> bunch of logging related changes around then and subsequently, so I
>> thought I'd mention it in case it does signal an issue (it may
>> alternatively indicate a need for increasing the open files limit).
>> FWIW, the build passed locally when I tried and seems as happy as
>> usual on the ASF Jenkins nodes.
>>
>> Robbie
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: Test run issues on qpid-java trunk?

[Keith W <ke...@gmail.com>]

Hi Robbie

QPID-6613 has been making some changes to test logging.  One area of change
is the generation of per-test log files which is now handled by Logback's
SiftingAppender.

I made a commit (r1690741) on Monday (13th) (to help investigate a
seemingly new sporadical issue where the Java Broker was failing to close
down completely because the Selector-Port thread remains running).  This
commit would mean the JVM running the tests would hold on to more file
handles for longer.  I have reverted this change as I expect that it is
contributing to the issue you are seeing.  I have just reverted this change.

I wonder if the issue you were observing from around 2nd/3rd July is the
same as my Selector-Port thread issue.  If you see a recurrence, can you
post some details?  I am having problems reproducing on the kit I have
available to me.

cheers, Keith.


On 15 July 2015 at 15:24, Robbie Gemmell <ro...@gmail.com> wrote:

> Hi folks,
>
> Irina mentioned seeing some failures for the java build in a CI system
> she runs. It has been failing consistently on one box, and
> intermittent on another. Taking a quick peek it seems like the root
> cause are the "Too many open files" exceptions happening at various
> points. One example is failing to open a Selector, another is Logback
> failing to create a file for its appender during setUp, and another
> was failing to find the class under test. The failures start part way
> through the broker-core module.
>
> It isn't clear exactly when the issue started as some of the
> intervening reports have already been removed, but it seems like the
> last successfull build was from July 2nd, and the first with issues
> *may* have been on July 3rd as a result. There looks to have been a
> bunch of logging related changes around then and subsequently, so I
> thought I'd mention it in case it does signal an issue (it may
> alternatively indicate a need for increasing the open files limit).
> FWIW, the build passed locally when I tried and seems as happy as
> usual on the ASF Jenkins nodes.
>
> Robbie
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>