You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by Martin Ritchie <ri...@apache.org> on 2010/06/03 15:22:02 UTC
Re: [Java] Repeated test failures in broker and broker pluggin
modules
On 3 June 2010 14:13, Rajith Attapattu <ra...@gmail.com> wrote:
> I have noticed repeated test failures in the broker and broker pluggin modules.
>
> While I am happy about the responsive of the community in general in
> fixing these issues, I am concerned about the frequency of these test
> failures.
> Repeated failures will undermine the confidence in the tests and end
> product, whether it's a genuine failure or due to an error in the test
> itself.
> (It should be noted, here I am only referring to failures in the
> default test profile and the java test profiles, but some of these
> failures are also affecting the cpp test profiles as well - all though
> that is a different topic all together.)
>
> Perhaps we need to be more careful when introducing new tests and also
> make sure the existing tests are consistently passing before adding
> any more.
> If the builds are red for prolonged periods then we run the risk of
> missing out on genuine test failures.
I am working on these Rajith, will have them fixed directly.
> Regards,
>
> Rajith Attapattu
> Red Hat
> http://rajith.2rlabs.com/
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
--
Martin Ritchie
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin modules
Posted by Gordon Sim <gs...@redhat.com>.
On 08/24/2010 04:45 PM, Marnie McCormack wrote:
> Is this still happening on CI regularly ?
No, haven't seen it for a while now.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin modules
Posted by Marnie McCormack <ma...@googlemail.com>.
Is this still happening on CI regularly ?
Thanks,
Marnie
On Fri, Jun 18, 2010 at 9:35 AM, Gordon Sim <gs...@redhat.com> wrote:
> On 06/18/2010 09:23 AM, Martin Ritchie wrote:
>
>> On 17 June 2010 14:24, Rajith Attapattu<ra...@gmail.com> wrote:
>>
>>> Martin,
>>>
>>> The tests are run using the default memory config given in the build.xml
>>> Java version is "java version "1.6.0_13" "
>>> OS is RHEL5 and we run on both 32& 64 bit machines.
>>>
>>> Apart from that I don't think there is anything special.
>>>
>>> Regards,
>>>
>>> Rajith
>>>
>>
>> Is there one particular profile that locks up more than the others?
>> Let me know and I'll leave it running on my desktop all weekend.
>> Locking up your CI boxes is not ideal so would be good to get this
>> resolved.
>>
>
> java.testprofile
>
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
Re: [Java] Repeated test failures in broker and broker pluggin modules
Posted by Gordon Sim <gs...@redhat.com>.
On 06/18/2010 09:23 AM, Martin Ritchie wrote:
> On 17 June 2010 14:24, Rajith Attapattu<ra...@gmail.com> wrote:
>> Martin,
>>
>> The tests are run using the default memory config given in the build.xml
>> Java version is "java version "1.6.0_13" "
>> OS is RHEL5 and we run on both 32& 64 bit machines.
>>
>> Apart from that I don't think there is anything special.
>>
>> Regards,
>>
>> Rajith
>
> Is there one particular profile that locks up more than the others?
> Let me know and I'll leave it running on my desktop all weekend.
> Locking up your CI boxes is not ideal so would be good to get this
> resolved.
java.testprofile
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin
modules
Posted by Martin Ritchie <ri...@apache.org>.
On 17 June 2010 14:24, Rajith Attapattu <ra...@gmail.com> wrote:
> Martin,
>
> The tests are run using the default memory config given in the build.xml
> Java version is "java version "1.6.0_13" "
> OS is RHEL5 and we run on both 32 & 64 bit machines.
>
> Apart from that I don't think there is anything special.
>
> Regards,
>
> Rajith
Is there one particular profile that locks up more than the others?
Let me know and I'll leave it running on my desktop all weekend.
Locking up your CI boxes is not ideal so would be good to get this
resolved.
Martin
> On Thu, Jun 17, 2010 at 8:16 AM, Martin Ritchie <ri...@apache.org> wrote:
>> On 17 June 2010 09:44, Gordon Sim <gs...@redhat.com> wrote:
>>> On 06/08/2010 04:55 PM, Gordon Sim wrote:
>>>>
>>>> On 06/08/2010 02:31 PM, Andrew Kennedy wrote:
>>>>>
>>>>> On 8 June 2010 08:32, Gordon Sim<gs...@redhat.com> wrote:
>>>>>>
>>>>>> "main" prio=10 tid=0x09a17800 nid=0x594d in Object.wait()
>>>>>> [0xb7ee8000..0xb7ee91f8]
>>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>>> at java.lang.Object.wait(Native Method)
>>>>>> at java.lang.Object.wait(Object.java:485)
>>>>>> at
>>>>>>
>>>>>> org.apache.felix.framework.StartLevelImpl.setStartLevelAndWait(StartLevelImpl.java:129)
>>>>>>
>>>>>> - locked<0x7767e508> (a java.lang.Integer)
>>>>>> at org.apache.felix.framework.Felix.start(Felix.java:762)
>>>>>> at
>>>>>>
>>>>>> org.apache.qpid.server.plugins.PluginManager.<init>(PluginManager.java:199)
>>>>>>
>>>>>
>>>>> I haven't seen this causing a hang before - this seems to be in the
>>>>> first test that actually uses the Felix framework for plugins
>>>>> (ExtraTest) causing this - do the tests consistently stop here?
>>>>
>>>> No it seems intermittent.
>>>
>>> Still seeing these quite frequently.
>>
>> Gordon,
>>
>> Can you provide more details on how your CI is setup? I haven't seen
>> this issue locally or on the variety of CI profiles that run 24/7.
>>
>> Cheers
>> Martin
>>> ---------------------------------------------------------------------
>>> Apache Qpid - AMQP Messaging Implementation
>>> Project: http://qpid.apache.org
>>> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>>>
>>>
>>
>>
>>
>> --
>> Martin Ritchie
>>
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>>
>>
>
>
>
> --
> Regards,
>
> Rajith Attapattu
> Red Hat
> http://rajith.2rlabs.com/
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
--
Martin Ritchie
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin
modules
Posted by Rajith Attapattu <ra...@gmail.com>.
Martin,
The tests are run using the default memory config given in the build.xml
Java version is "java version "1.6.0_13" "
OS is RHEL5 and we run on both 32 & 64 bit machines.
Apart from that I don't think there is anything special.
Regards,
Rajith
On Thu, Jun 17, 2010 at 8:16 AM, Martin Ritchie <ri...@apache.org> wrote:
> On 17 June 2010 09:44, Gordon Sim <gs...@redhat.com> wrote:
>> On 06/08/2010 04:55 PM, Gordon Sim wrote:
>>>
>>> On 06/08/2010 02:31 PM, Andrew Kennedy wrote:
>>>>
>>>> On 8 June 2010 08:32, Gordon Sim<gs...@redhat.com> wrote:
>>>>>
>>>>> "main" prio=10 tid=0x09a17800 nid=0x594d in Object.wait()
>>>>> [0xb7ee8000..0xb7ee91f8]
>>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>>> at java.lang.Object.wait(Native Method)
>>>>> at java.lang.Object.wait(Object.java:485)
>>>>> at
>>>>>
>>>>> org.apache.felix.framework.StartLevelImpl.setStartLevelAndWait(StartLevelImpl.java:129)
>>>>>
>>>>> - locked<0x7767e508> (a java.lang.Integer)
>>>>> at org.apache.felix.framework.Felix.start(Felix.java:762)
>>>>> at
>>>>>
>>>>> org.apache.qpid.server.plugins.PluginManager.<init>(PluginManager.java:199)
>>>>>
>>>>
>>>> I haven't seen this causing a hang before - this seems to be in the
>>>> first test that actually uses the Felix framework for plugins
>>>> (ExtraTest) causing this - do the tests consistently stop here?
>>>
>>> No it seems intermittent.
>>
>> Still seeing these quite frequently.
>
> Gordon,
>
> Can you provide more details on how your CI is setup? I haven't seen
> this issue locally or on the variety of CI profiles that run 24/7.
>
> Cheers
> Martin
>> ---------------------------------------------------------------------
>> Apache Qpid - AMQP Messaging Implementation
>> Project: http://qpid.apache.org
>> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>>
>>
>
>
>
> --
> Martin Ritchie
>
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
--
Regards,
Rajith Attapattu
Red Hat
http://rajith.2rlabs.com/
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin
modules
Posted by Martin Ritchie <ri...@apache.org>.
On 17 June 2010 09:44, Gordon Sim <gs...@redhat.com> wrote:
> On 06/08/2010 04:55 PM, Gordon Sim wrote:
>>
>> On 06/08/2010 02:31 PM, Andrew Kennedy wrote:
>>>
>>> On 8 June 2010 08:32, Gordon Sim<gs...@redhat.com> wrote:
>>>>
>>>> "main" prio=10 tid=0x09a17800 nid=0x594d in Object.wait()
>>>> [0xb7ee8000..0xb7ee91f8]
>>>> java.lang.Thread.State: WAITING (on object monitor)
>>>> at java.lang.Object.wait(Native Method)
>>>> at java.lang.Object.wait(Object.java:485)
>>>> at
>>>>
>>>> org.apache.felix.framework.StartLevelImpl.setStartLevelAndWait(StartLevelImpl.java:129)
>>>>
>>>> - locked<0x7767e508> (a java.lang.Integer)
>>>> at org.apache.felix.framework.Felix.start(Felix.java:762)
>>>> at
>>>>
>>>> org.apache.qpid.server.plugins.PluginManager.<init>(PluginManager.java:199)
>>>>
>>>
>>> I haven't seen this causing a hang before - this seems to be in the
>>> first test that actually uses the Felix framework for plugins
>>> (ExtraTest) causing this - do the tests consistently stop here?
>>
>> No it seems intermittent.
>
> Still seeing these quite frequently.
Gordon,
Can you provide more details on how your CI is setup? I haven't seen
this issue locally or on the variety of CI profiles that run 24/7.
Cheers
Martin
> ---------------------------------------------------------------------
> Apache Qpid - AMQP Messaging Implementation
> Project: http://qpid.apache.org
> Use/Interact: mailto:dev-subscribe@qpid.apache.org
>
>
--
Martin Ritchie
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin modules
Posted by Gordon Sim <gs...@redhat.com>.
On 06/08/2010 04:55 PM, Gordon Sim wrote:
> On 06/08/2010 02:31 PM, Andrew Kennedy wrote:
>> On 8 June 2010 08:32, Gordon Sim<gs...@redhat.com> wrote:
>>> "main" prio=10 tid=0x09a17800 nid=0x594d in Object.wait()
>>> [0xb7ee8000..0xb7ee91f8]
>>> java.lang.Thread.State: WAITING (on object monitor)
>>> at java.lang.Object.wait(Native Method)
>>> at java.lang.Object.wait(Object.java:485)
>>> at
>>> org.apache.felix.framework.StartLevelImpl.setStartLevelAndWait(StartLevelImpl.java:129)
>>>
>>> - locked<0x7767e508> (a java.lang.Integer)
>>> at org.apache.felix.framework.Felix.start(Felix.java:762)
>>> at
>>> org.apache.qpid.server.plugins.PluginManager.<init>(PluginManager.java:199)
>>>
>>
>> I haven't seen this causing a hang before - this seems to be in the
>> first test that actually uses the Felix framework for plugins
>> (ExtraTest) causing this - do the tests consistently stop here?
>
> No it seems intermittent.
Still seeing these quite frequently.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin modules
Posted by Gordon Sim <gs...@redhat.com>.
On 06/08/2010 02:31 PM, Andrew Kennedy wrote:
> On 8 June 2010 08:32, Gordon Sim<gs...@redhat.com> wrote:
>> "main" prio=10 tid=0x09a17800 nid=0x594d in Object.wait() [0xb7ee8000..0xb7ee91f8]
>> java.lang.Thread.State: WAITING (on object monitor)
>> at java.lang.Object.wait(Native Method)
>> at java.lang.Object.wait(Object.java:485)
>> at org.apache.felix.framework.StartLevelImpl.setStartLevelAndWait(StartLevelImpl.java:129)
>> - locked<0x7767e508> (a java.lang.Integer)
>> at org.apache.felix.framework.Felix.start(Felix.java:762)
>> at org.apache.qpid.server.plugins.PluginManager.<init>(PluginManager.java:199)
>
> I haven't seen this causing a hang before - this seems to be in the
> first test that actually uses the Felix framework for plugins
> (ExtraTest) causing this - do the tests consistently stop here?
No it seems intermittent.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin
modules
Posted by Andrew Kennedy <an...@gmail.com>.
On 8 June 2010 08:32, Gordon Sim <gs...@redhat.com> wrote:
> "main" prio=10 tid=0x09a17800 nid=0x594d in Object.wait() [0xb7ee8000..0xb7ee91f8]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.felix.framework.StartLevelImpl.setStartLevelAndWait(StartLevelImpl.java:129)
> - locked <0x7767e508> (a java.lang.Integer)
> at org.apache.felix.framework.Felix.start(Felix.java:762)
> at org.apache.qpid.server.plugins.PluginManager.<init>(PluginManager.java:199)
I haven't seen this causing a hang before - this seems to be in the
first test that actually uses the Felix framework for plugins
(ExtraTest) causing this - do the tests consistently stop here?
Andrew.
--
-- andrew d kennedy ? edinburgh : +44 7941 197 134
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org
Re: [Java] Repeated test failures in broker and broker pluggin modules
Posted by Gordon Sim <gs...@redhat.com>.
Anyone else seeing hangs in starting Felix? Is this a known issue? (This
is from an automated test run).
> "main" prio=10 tid=0x09a17800 nid=0x594d in Object.wait() [0xb7ee8000..0xb7ee91f8]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.felix.framework.StartLevelImpl.setStartLevelAndWait(StartLevelImpl.java:129)
> - locked <0x7767e508> (a java.lang.Integer)
> at org.apache.felix.framework.Felix.start(Felix.java:762)
> at org.apache.qpid.server.plugins.PluginManager.<init>(PluginManager.java:199)
> at org.apache.qpid.server.registry.ApplicationRegistry.configure(ApplicationRegistry.java:240)
> at org.apache.qpid.server.registry.ApplicationRegistry.initialise(ApplicationRegistry.java:258)
> at org.apache.qpid.server.registry.ApplicationRegistry.initialise(ApplicationRegistry.java:142)
> at org.apache.qpid.server.registry.ApplicationRegistry.initialise(ApplicationRegistry.java:119)
> at org.apache.qpid.server.plugins.ExtrasTest.setUp(ExtrasTest.java:49)
> at junit.framework.TestCase.runBare(TestCase.java:128)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:120)
> at junit.framework.TestSuite.runTest(TestSuite.java:230)
> at junit.framework.TestSuite.run(TestSuite.java:225)
> at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:297)
> at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeInVM(JUnitTask.java:1072)
> at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:682)
> at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.executeOrQueue(JUnitTask.java:1434)
> at org.apache.tools.ant.taskdefs.optional.junit.JUnitTask.execute(JUnitTask.java:633)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
> at org.apache.tools.ant.Task.perform(Task.java:364)
> at org.apache.tools.ant.Target.execute(Target.java:341)
> at org.apache.tools.ant.Target.performTasks(Target.java:369)
> at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216)
> at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:37)
> at org.apache.tools.ant.Project.executeTargets(Project.java:1068)
> at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:382)
> at org.apache.tools.ant.taskdefs.SubAnt.execute(SubAnt.java:277)
> at org.apache.tools.ant.taskdefs.SubAnt.execute(SubAnt.java:201)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
> at org.apache.tools.ant.Task.perform(Task.java:364)
> at org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:64)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
> at org.apache.tools.ant.Task.perform(Task.java:364)
> at org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:378)
> at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:275)
> at org.apache.tools.ant.Task.perform(Task.java:364)
> at org.apache.tools.ant.Target.execute(Target.java:341)
> at org.apache.tools.ant.Target.performTasks(Target.java:369)
> at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1216)
> at org.apache.tools.ant.Project.executeTarget(Project.java:1185)
> at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:40)
> at org.apache.tools.ant.Project.executeTargets(Project.java:1068)
> at org.apache.tools.ant.Main.runBuild(Main.java:668)
> at org.apache.tools.ant.Main.startAnt(Main.java:187)
> at org.apache.tools.ant.launch.Launcher.run(Launcher.java:246)
> at org.apache.tools.ant.launch.Launcher.main(Launcher.java:67)
>
There are literally a hundred or so threads in the stack dumps, nothing
else standing out. The following ar ethe 'felix' threads in case that is
useful:
> "FelixPackageAdmin" daemon prio=10 tid=0x6c036000 nid=0x7826 in Object.wait() [0x6a706000..0x6a706fb0]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.felix.framework.PackageAdminImpl.run(PackageAdminImpl.java:316)
> - locked <0x7767a628> (a org.apache.felix.framework.PackageAdminImpl)
> at java.lang.Thread.run(Thread.java:619)
>
> "FelixPackageAdmin" daemon prio=10 tid=0x0a8cf000 nid=0x7824 in Object.wait() [0x6f532000..0x6f5330b0]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.felix.framework.PackageAdminImpl.run(PackageAdminImpl.java:316)
> - locked <0x77176528> (a org.apache.felix.framework.PackageAdminImpl)
> at java.lang.Thread.run(Thread.java:619)
>
> "FelixStartLevel" daemon prio=10 tid=0x0a8ce800 nid=0x7823 in Object.wait() [0x6fc5c000..0x6fc5ce30]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:242)
> - locked <0x77188768> (a java.util.ArrayList)
> at java.lang.Thread.run(Thread.java:619)
>
> "FelixDispatchQueue" prio=10 tid=0x09f01400 nid=0x7822 in Object.wait() [0x6cfcd000..0x6cfcddb0]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.felix.framework.util.EventDispatcher.run(EventDispatcher.java:917)
> - locked <0x771cdb48> (a java.util.ArrayList)
> at org.apache.felix.framework.util.EventDispatcher.access$000(EventDispatcher.java:54)
> at org.apache.felix.framework.util.EventDispatcher$1.run(EventDispatcher.java:106)
> at java.lang.Thread.run(Thread.java:619)
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:dev-subscribe@qpid.apache.org