You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Jonathan Gallimore <jo...@gmail.com> on 2021/03/12 13:29:42 UTC
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
I ran into this test failing yesterday:
activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
- I'd be happy to try and contribute a fix. Would you like to assign the
JIRA to me?
Jon
On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
wrote:
> Hi guys,
>
> Now that we have Jenkinsfile in our repo, and we use Jenkins pipeline, we
> dramatically improved our build: the build is executed for each
> PullRequests or commit on the main branch.
>
> However, we have lot of failing tests, causing quite systematically the
> build failing on ci-builds.apache.org.
>
> We really need to have a clean, accurate and stable build: it will improve
> the issue detection and simplify the review, especially for PullRequests.
>
> I ran several builds on my machine (with different docker containers) and
> I already identified some failing/flaky tests:
>
> -
> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
> is not a big deal as I have a PR removing leveled completely
> -
> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
> Chris did an improvement, but I still have some flakiness here.
> -
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>
> I propose the following action plan:
>
> 1. Create the Jira for each failing/flaky tests
> 2. Exclude the tests (in surefire plugin configuration) to have a "green
> light" on Jenkins.
> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins is
> still "happy".
>
> Anyone willing to help on (3) is welcome !
>
> If there’s no objection, I will start with (1) and (2).
>
> Thanks,
> Regards
> JB
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by JB Onofré <jb...@nanthrax.net>.
My point is just to exclude temporary and to fix them ASAP.
So it’s just to have a clean build: either excluding them or fixing them ;)
Regards
JB
> Le 12 mars 2021 à 19:26, Christopher Shannon <ch...@gmail.com> a écrit :
>
> Ideally it would be better to fix tests than to simply exclude them. These
> tests were added for a reason I would presume (I know I had worked on the
> durable sync stuff in the past) so randomly turning off tests could lead to
> missing errors.
>
>
>
>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>> wrote:
>>
>> I’m adding these tests to be fixed/improved:
>>
>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>
>> Let me create the Jira and create a PR to exclude the tests and verify
>> Jenkins is happy.
>>
>> Regards
>> JB
>>
>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>> jonathan.gallimore@gmail.com> a écrit :
>>>
>>> I'm +1 on the actions :).
>>>
>>> Jon
>>>
>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Sure, thanks for the help !
>>>>
>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>
>>>> Regards
>>>> JB
>>>>
>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>
>>>>> I ran into this test failing yesterday:
>>>>>
>>>>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>> the
>>>>> JIRA to me?
>>>>>
>>>>> Jon
>>>>>
>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>
>>>>> wrote:
>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins pipeline,
>>>> we
>>>>>> dramatically improved our build: the build is executed for each
>>>>>> PullRequests or commit on the main branch.
>>>>>>
>>>>>> However, we have lot of failing tests, causing quite systematically
>> the
>>>>>> build failing on ci-builds.apache.org.
>>>>>>
>>>>>> We really need to have a clean, accurate and stable build: it will
>>>> improve
>>>>>> the issue detection and simplify the review, especially for
>>>> PullRequests.
>>>>>>
>>>>>> I ran several builds on my machine (with different docker containers)
>>>> and
>>>>>> I already identified some failing/flaky tests:
>>>>>>
>>>>>> -
>>>>>>
>>>>
>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>> -
>>>>>>
>>>>
>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>> -
>>>>>>
>>>>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>
>>>>>> I propose the following action plan:
>>>>>>
>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>> "green
>>>>>> light" on Jenkins.
>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins is
>>>>>> still "happy".
>>>>>>
>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>
>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>
>>>>>> Thanks,
>>>>>> Regards
>>>>>> JB
>>>>
>>>>
>>
>>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi Etienne,
Thanks for your feedback.
Yes, regarding how long is the build, even on PR, we are on a "challenging" path (2 hours to build each PR is long, and that’s why it should be stable).
A full build takes between 3 or 4 hours, so, definitely not possible on PR. That’s why I proposed a run once a week (it’s not nightly build, it’s a weekly build ;)), generating SNAPSHOTs.
We can always run this weekly build on demand (via ci-builds.apache.org <http://ci-builds.apache.org/>).
I can include on this "weekly build" the versions checker generation (I have a profile a local branch doing that).
Regards
JB
> Le 16 mars 2021 à 21:24, Hossack, Etienne <eh...@amazon.com.INVALID> a écrit :
>
> Hi all, just some thoughts to share here:
>
> I think ideally I would expect a “static” build to be a nightly build run every day - but I think given the frequency of contributions, weekly makes sense (and I know it takes about 30% of the day 😅).
> Maybe too, that runs a snapshot version of the dependency checker to fail the build on CVEs or new types of checks from that tool.
>
> And then for contributions, the PR can run the lightweight profile, but then master could run the full profile on merge?
>
> Does that make sense?
> In summary I think it’s me expressing agreement for a static build, but also suggesting a full build be run on contributions in case there are multiple merges in a week, or say right after the build is run, and increasing the time-to-discovery of errors.
>
> Cheers,
> Étienne Hossack
> Software Development Engineer, Amazon MQ
> email: ehossack@amazon.com <ma...@amazon.com>
> phone: +1-778-945-8287
>
>
>
>> On Mar 15, 2021, at 10:05 PM, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>
>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>
>>
>>
>> Hi guys,
>>
>> I created https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622> PR about this and you can see that Jenkins is happy now. The full build took about 120mn (2h) on Jenkins.
>>
>> Basically what I did in the PR:
>> - remove activemq-unit-tests and itests (Karaf, Spring3) from the default reactor
>> - introduce full.test profile that build all modules including unit tests and itests
>>
>> The full.test profile is not use in Jenkinsfile, meaning that the PR executes all tests but not activemq-unit-tests modules neither itests. I think it’s acceptable for PR (and it already takes 2 hours ;)).
>> I would like to introduce a "static" build on ci-builds.apache.org <http://ci-builds.apache.org/> (not via Jenkinsfile) executed every week and doing a full build (including full.test profile).
>>
>> Thoughts ?
>>
>> Regards
>> JB
>>
>>> Le 15 mars 2021 à 08:20, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>
>>> Hi guys,
>>>
>>> I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
>>>
>>> AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
>>> AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
>>> AMQ-8188: AMQ5266SingleDestTest is failing
>>>
>>> There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593>).
>>>
>>> I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
>>>
>>> I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
>>>
>>> AMQ-8191 is almost ready, I’m testing.
>>>
>>> Regards
>>> JB
>>>
>>>> Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>>
>>>> Hi guys,
>>>>
>>>> I’ve updated my local branch according to your comments:
>>>>
>>>> 1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
>>>> 2. I’m keeping the smoke test profile
>>>> 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
>>>> 4. I’ve updated Jenkinsfile to use fast profile on PR
>>>>
>>>> I will create the PR soon.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>>> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>>>
>>>>> Hi,
>>>>>
>>>>> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>>>>>
>>>>> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>>>>>
>>>>> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>>>>>
>>>>> Thanks !
>>>>> I will improve this way.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> Le 12 mars 2021 à 20:31, Clebert Suconic <clebert.suconic@gmail.com <ma...@gmail.com>> a écrit :
>>>>>>
>>>>>> You should instead have a fast profile, with a subset of the testsuite
>>>>>> to run on every commit and branch for these cases. I looked on Jenkins
>>>>>> and having many builds taking 3 Hours each won't really scale on the
>>>>>> lab anyway. Failures will only make things worse there.
>>>>>>
>>>>>> The lab is usually not powerful for long running tests.
>>>>>>
>>>>>> And a full profile that should run as part of a full run. (say.. once
>>>>>> a day instead of every commit), or any interval you chose.
>>>>>>
>>>>>> I don't think you should hide tests though.. as that is like pushing
>>>>>> dirt under the rug.. (even if you say to enable it later... as in
>>>>>> anything in life temporary solutions endup being definitive usually).
>>>>>>
>>>>>> As any System dealing with times and asynchronous flaky and races are
>>>>>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>>>>>> Rule where the test is retried. You could also add retries to tests in
>>>>>> cases where it is acceptable... but be careful to not just hide bugs
>>>>>> away in this case as well.
>>>>>>
>>>>>> If you are interested, on artemis, Look for usages on
>>>>>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java>
>>>>>>
>>>>>>
>>>>>> You need to activate a profile in artemis for the retryRule to work.
>>>>>>
>>>>>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>
>>>>>>> Yes agree. I’m launching new builds ;)
>>>>>>>
>>>>>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>>>>>
>>>>>>>> Just running it by itself on the command line and also in the IDE. The full
>>>>>>>> build takes a while and if it's breaking with that then it's probably some
>>>>>>>> other test that isn't cleaning up properly in between runs.
>>>>>>>>
>>>>>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>>
>>>>>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>>>>>> build.
>>>>>>>>>
>>>>>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>>>>>> box
>>>>>>>>>> and it always passes.
>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>>>>>> These
>>>>>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>>>>>> the
>>>>>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>>>>>> lead to
>>>>>>>>>>> missing errors.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>>>>>
>>>>>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>>>>>
>>>>>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>>>>>> Jenkins is happy.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> JB
>>>>>>>>>>>>
>>>>>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>>>>>> pipeline,
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>>>>>> improve
>>>>>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>>>>>> PullRequests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>>>>>> containers)
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>>>>>> "green
>>>>>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Clebert Suconic
>>>>>
>>>>
>>>
>>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Matt Pavlovich <ma...@gmail.com>.
+1
> On Mar 17, 2021, at 12:00 AM, Jean-Baptiste Onofre <jb...@nanthrax.net> wrote:
>
> By the way, guys, I would like to merge:
>
> https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622>
>
> And then rebase PR branches to have "Jenkins happy".
>
> No objection to merge PR #622 ?
>
> Thanks,
> Regards
> JB
>
>> Le 17 mars 2021 à 05:45, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>>
>> Hi Matt,
>>
>> I agree.
>>
>> I think we should do a full build not after a single merge, but a "group" of merges. Else, it means that we will do a full build after each PR merge, so basically it’s what we have today, and not practical at all.
>>
>> That’s why, as a first step, I’m proposing to run once a week or "on demand".
>>
>> Regards
>> JB
>>
>>> Le 16 mars 2021 à 22:15, Matt Pavlovich <ma...@gmail.com> a écrit :
>>>
>>> Feels like we are in a transition period. I don’t see a per-PR unit test job being practical until the execution times come way down— and that is going to be significant engineering effort.
>>>
>>> That being said, full build with full tests the day after a merged change seems like a reasonable schedule.
>>>
>>>> On Mar 16, 2021, at 3:24 PM, Hossack, Etienne <ehossack@amazon.com.INVALID <ma...@amazon.com.INVALID>> wrote:
>>>>
>>>> Hi all, just some thoughts to share here:
>>>>
>>>> I think ideally I would expect a “static” build to be a nightly build run every day - but I think given the frequency of contributions, weekly makes sense (and I know it takes about 30% of the day 😅).
>>>> Maybe too, that runs a snapshot version of the dependency checker to fail the build on CVEs or new types of checks from that tool.
>>>>
>>>> And then for contributions, the PR can run the lightweight profile, but then master could run the full profile on merge?
>>>>
>>>> Does that make sense?
>>>> In summary I think it’s me expressing agreement for a static build, but also suggesting a full build be run on contributions in case there are multiple merges in a week, or say right after the build is run, and increasing the time-to-discovery of errors.
>>>>
>>>> Cheers,
>>>> Étienne Hossack
>>>> Software Development Engineer, Amazon MQ
>>>> email: ehossack@amazon.com <ma...@amazon.com> <mailto:ehossack@amazon.com <ma...@amazon.com>>
>>>> phone: +1-778-945-8287
>>>>
>>>>
>>>>
>>>>> On Mar 15, 2021, at 10:05 PM, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>>>>>
>>>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>>>
>>>>>
>>>>>
>>>>> Hi guys,
>>>>>
>>>>> I created https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622> <https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622>> PR about this and you can see that Jenkins is happy now. The full build took about 120mn (2h) on Jenkins.
>>>>>
>>>>> Basically what I did in the PR:
>>>>> - remove activemq-unit-tests and itests (Karaf, Spring3) from the default reactor
>>>>> - introduce full.test profile that build all modules including unit tests and itests
>>>>>
>>>>> The full.test profile is not use in Jenkinsfile, meaning that the PR executes all tests but not activemq-unit-tests modules neither itests. I think it’s acceptable for PR (and it already takes 2 hours ;)).
>>>>> I would like to introduce a "static" build on ci-builds.apache.org <http://ci-builds.apache.org/> <http://ci-builds.apache.org/ <http://ci-builds.apache.org/>> (not via Jenkinsfile) executed every week and doing a full build (including full.test profile).
>>>>>
>>>>> Thoughts ?
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> Le 15 mars 2021 à 08:20, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
>>>>>>
>>>>>> AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
>>>>>> AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
>>>>>> AMQ-8188: AMQ5266SingleDestTest is failing
>>>>>>
>>>>>> There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593> <https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593>>).
>>>>>>
>>>>>> I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
>>>>>>
>>>>>> I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
>>>>>>
>>>>>> AMQ-8191 is almost ready, I’m testing.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>>> Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>>>
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> I’ve updated my local branch according to your comments:
>>>>>>>
>>>>>>> 1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
>>>>>>> 2. I’m keeping the smoke test profile
>>>>>>> 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
>>>>>>> 4. I’ve updated Jenkinsfile to use fast profile on PR
>>>>>>>
>>>>>>> I will create the PR soon.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>>> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>>>>>>>>
>>>>>>>> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>>>>>>>>
>>>>>>>> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>>>>>>>>
>>>>>>>> Thanks !
>>>>>>>> I will improve this way.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> JB
>>>>>>>>
>>>>>>>>> Le 12 mars 2021 à 20:31, Clebert Suconic <clebert.suconic@gmail.com <ma...@gmail.com> <mailto:clebert.suconic@gmail.com <ma...@gmail.com>>> a écrit :
>>>>>>>>>
>>>>>>>>> You should instead have a fast profile, with a subset of the testsuite
>>>>>>>>> to run on every commit and branch for these cases. I looked on Jenkins
>>>>>>>>> and having many builds taking 3 Hours each won't really scale on the
>>>>>>>>> lab anyway. Failures will only make things worse there.
>>>>>>>>>
>>>>>>>>> The lab is usually not powerful for long running tests.
>>>>>>>>>
>>>>>>>>> And a full profile that should run as part of a full run. (say.. once
>>>>>>>>> a day instead of every commit), or any interval you chose.
>>>>>>>>>
>>>>>>>>> I don't think you should hide tests though.. as that is like pushing
>>>>>>>>> dirt under the rug.. (even if you say to enable it later... as in
>>>>>>>>> anything in life temporary solutions endup being definitive usually).
>>>>>>>>>
>>>>>>>>> As any System dealing with times and asynchronous flaky and races are
>>>>>>>>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>>>>>>>>> Rule where the test is retried. You could also add retries to tests in
>>>>>>>>> cases where it is acceptable... but be careful to not just hide bugs
>>>>>>>>> away in this case as well.
>>>>>>>>>
>>>>>>>>> If you are interested, on artemis, Look for usages on
>>>>>>>>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java> <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You need to activate a profile in artemis for the retryRule to work.
>>>>>>>>>
>>>>>>>>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>>>
>>>>>>>>>> Yes agree. I’m launching new builds ;)
>>>>>>>>>>
>>>>>>>>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>> Just running it by itself on the command line and also in the IDE. The full
>>>>>>>>>>> build takes a while and if it's breaking with that then it's probably some
>>>>>>>>>>> other test that isn't cleaning up properly in between runs.
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>>>>>>>>> build.
>>>>>>>>>>>>
>>>>>>>>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>>>>>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>>>>>>>>> box
>>>>>>>>>>>>> and it always passes.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>>>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>>>>>>>>> These
>>>>>>>>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>>>>>>>>> the
>>>>>>>>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>>>>>>>>> lead to
>>>>>>>>>>>>>> missing errors.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>>>>>>>>> Jenkins is happy.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>>>>>>>>> pipeline,
>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>>>>>>>>> improve
>>>>>>>>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>>>>>>>>> PullRequests.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>>>>>>>>> containers)
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>>>>>>>>> "green
>>>>>>>>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Clebert Suconic
>>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
By the way, guys, I would like to merge:
https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622>
And then rebase PR branches to have "Jenkins happy".
No objection to merge PR #622 ?
Thanks,
Regards
JB
> Le 17 mars 2021 à 05:45, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>
> Hi Matt,
>
> I agree.
>
> I think we should do a full build not after a single merge, but a "group" of merges. Else, it means that we will do a full build after each PR merge, so basically it’s what we have today, and not practical at all.
>
> That’s why, as a first step, I’m proposing to run once a week or "on demand".
>
> Regards
> JB
>
>> Le 16 mars 2021 à 22:15, Matt Pavlovich <ma...@gmail.com> a écrit :
>>
>> Feels like we are in a transition period. I don’t see a per-PR unit test job being practical until the execution times come way down— and that is going to be significant engineering effort.
>>
>> That being said, full build with full tests the day after a merged change seems like a reasonable schedule.
>>
>>> On Mar 16, 2021, at 3:24 PM, Hossack, Etienne <ehossack@amazon.com.INVALID <ma...@amazon.com.INVALID>> wrote:
>>>
>>> Hi all, just some thoughts to share here:
>>>
>>> I think ideally I would expect a “static” build to be a nightly build run every day - but I think given the frequency of contributions, weekly makes sense (and I know it takes about 30% of the day 😅).
>>> Maybe too, that runs a snapshot version of the dependency checker to fail the build on CVEs or new types of checks from that tool.
>>>
>>> And then for contributions, the PR can run the lightweight profile, but then master could run the full profile on merge?
>>>
>>> Does that make sense?
>>> In summary I think it’s me expressing agreement for a static build, but also suggesting a full build be run on contributions in case there are multiple merges in a week, or say right after the build is run, and increasing the time-to-discovery of errors.
>>>
>>> Cheers,
>>> Étienne Hossack
>>> Software Development Engineer, Amazon MQ
>>> email: ehossack@amazon.com <ma...@amazon.com> <mailto:ehossack@amazon.com <ma...@amazon.com>>
>>> phone: +1-778-945-8287
>>>
>>>
>>>
>>>> On Mar 15, 2021, at 10:05 PM, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>>>>
>>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>>
>>>>
>>>>
>>>> Hi guys,
>>>>
>>>> I created https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622> <https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622>> PR about this and you can see that Jenkins is happy now. The full build took about 120mn (2h) on Jenkins.
>>>>
>>>> Basically what I did in the PR:
>>>> - remove activemq-unit-tests and itests (Karaf, Spring3) from the default reactor
>>>> - introduce full.test profile that build all modules including unit tests and itests
>>>>
>>>> The full.test profile is not use in Jenkinsfile, meaning that the PR executes all tests but not activemq-unit-tests modules neither itests. I think it’s acceptable for PR (and it already takes 2 hours ;)).
>>>> I would like to introduce a "static" build on ci-builds.apache.org <http://ci-builds.apache.org/> <http://ci-builds.apache.org/ <http://ci-builds.apache.org/>> (not via Jenkinsfile) executed every week and doing a full build (including full.test profile).
>>>>
>>>> Thoughts ?
>>>>
>>>> Regards
>>>> JB
>>>>
>>>>> Le 15 mars 2021 à 08:20, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>
>>>>> Hi guys,
>>>>>
>>>>> I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
>>>>>
>>>>> AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
>>>>> AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
>>>>> AMQ-8188: AMQ5266SingleDestTest is failing
>>>>>
>>>>> There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593> <https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593>>).
>>>>>
>>>>> I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
>>>>>
>>>>> I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
>>>>>
>>>>> AMQ-8191 is almost ready, I’m testing.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>>
>>>>>> Hi guys,
>>>>>>
>>>>>> I’ve updated my local branch according to your comments:
>>>>>>
>>>>>> 1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
>>>>>> 2. I’m keeping the smoke test profile
>>>>>> 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
>>>>>> 4. I’ve updated Jenkinsfile to use fast profile on PR
>>>>>>
>>>>>> I will create the PR soon.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>>> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>>>>>>>
>>>>>>> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>>>>>>>
>>>>>>> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>>>>>>>
>>>>>>> Thanks !
>>>>>>> I will improve this way.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>>> Le 12 mars 2021 à 20:31, Clebert Suconic <clebert.suconic@gmail.com <ma...@gmail.com> <mailto:clebert.suconic@gmail.com <ma...@gmail.com>>> a écrit :
>>>>>>>>
>>>>>>>> You should instead have a fast profile, with a subset of the testsuite
>>>>>>>> to run on every commit and branch for these cases. I looked on Jenkins
>>>>>>>> and having many builds taking 3 Hours each won't really scale on the
>>>>>>>> lab anyway. Failures will only make things worse there.
>>>>>>>>
>>>>>>>> The lab is usually not powerful for long running tests.
>>>>>>>>
>>>>>>>> And a full profile that should run as part of a full run. (say.. once
>>>>>>>> a day instead of every commit), or any interval you chose.
>>>>>>>>
>>>>>>>> I don't think you should hide tests though.. as that is like pushing
>>>>>>>> dirt under the rug.. (even if you say to enable it later... as in
>>>>>>>> anything in life temporary solutions endup being definitive usually).
>>>>>>>>
>>>>>>>> As any System dealing with times and asynchronous flaky and races are
>>>>>>>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>>>>>>>> Rule where the test is retried. You could also add retries to tests in
>>>>>>>> cases where it is acceptable... but be careful to not just hide bugs
>>>>>>>> away in this case as well.
>>>>>>>>
>>>>>>>> If you are interested, on artemis, Look for usages on
>>>>>>>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java> <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java>>
>>>>>>>>
>>>>>>>>
>>>>>>>> You need to activate a profile in artemis for the retryRule to work.
>>>>>>>>
>>>>>>>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>>
>>>>>>>>> Yes agree. I’m launching new builds ;)
>>>>>>>>>
>>>>>>>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> Just running it by itself on the command line and also in the IDE. The full
>>>>>>>>>> build takes a while and if it's breaking with that then it's probably some
>>>>>>>>>> other test that isn't cleaning up properly in between runs.
>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>>>>>>>> build.
>>>>>>>>>>>
>>>>>>>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>>>>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>>>>>>>> box
>>>>>>>>>>>> and it always passes.
>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>>>>>>>> These
>>>>>>>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>>>>>>>> the
>>>>>>>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>>>>>>>> lead to
>>>>>>>>>>>>> missing errors.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>>>>>>>> Jenkins is happy.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>>>>>>>> pipeline,
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>>>>>>>> improve
>>>>>>>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>>>>>>>> PullRequests.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>>>>>>>> containers)
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>>>>>>>> "green
>>>>>>>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Clebert Suconic
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi Matt,
I agree.
I think we should do a full build not after a single merge, but a "group" of merges. Else, it means that we will do a full build after each PR merge, so basically it’s what we have today, and not practical at all.
That’s why, as a first step, I’m proposing to run once a week or "on demand".
Regards
JB
> Le 16 mars 2021 à 22:15, Matt Pavlovich <ma...@gmail.com> a écrit :
>
> Feels like we are in a transition period. I don’t see a per-PR unit test job being practical until the execution times come way down— and that is going to be significant engineering effort.
>
> That being said, full build with full tests the day after a merged change seems like a reasonable schedule.
>
>> On Mar 16, 2021, at 3:24 PM, Hossack, Etienne <ehossack@amazon.com.INVALID <ma...@amazon.com.INVALID>> wrote:
>>
>> Hi all, just some thoughts to share here:
>>
>> I think ideally I would expect a “static” build to be a nightly build run every day - but I think given the frequency of contributions, weekly makes sense (and I know it takes about 30% of the day 😅).
>> Maybe too, that runs a snapshot version of the dependency checker to fail the build on CVEs or new types of checks from that tool.
>>
>> And then for contributions, the PR can run the lightweight profile, but then master could run the full profile on merge?
>>
>> Does that make sense?
>> In summary I think it’s me expressing agreement for a static build, but also suggesting a full build be run on contributions in case there are multiple merges in a week, or say right after the build is run, and increasing the time-to-discovery of errors.
>>
>> Cheers,
>> Étienne Hossack
>> Software Development Engineer, Amazon MQ
>> email: ehossack@amazon.com <ma...@amazon.com> <mailto:ehossack@amazon.com <ma...@amazon.com>>
>> phone: +1-778-945-8287
>>
>>
>>
>>> On Mar 15, 2021, at 10:05 PM, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>>>
>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>>
>>>
>>>
>>> Hi guys,
>>>
>>> I created https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622> <https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622>> PR about this and you can see that Jenkins is happy now. The full build took about 120mn (2h) on Jenkins.
>>>
>>> Basically what I did in the PR:
>>> - remove activemq-unit-tests and itests (Karaf, Spring3) from the default reactor
>>> - introduce full.test profile that build all modules including unit tests and itests
>>>
>>> The full.test profile is not use in Jenkinsfile, meaning that the PR executes all tests but not activemq-unit-tests modules neither itests. I think it’s acceptable for PR (and it already takes 2 hours ;)).
>>> I would like to introduce a "static" build on ci-builds.apache.org <http://ci-builds.apache.org/> <http://ci-builds.apache.org/ <http://ci-builds.apache.org/>> (not via Jenkinsfile) executed every week and doing a full build (including full.test profile).
>>>
>>> Thoughts ?
>>>
>>> Regards
>>> JB
>>>
>>>> Le 15 mars 2021 à 08:20, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>
>>>> Hi guys,
>>>>
>>>> I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
>>>>
>>>> AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
>>>> AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
>>>> AMQ-8188: AMQ5266SingleDestTest is failing
>>>>
>>>> There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593> <https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593>>).
>>>>
>>>> I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
>>>>
>>>> I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
>>>>
>>>> AMQ-8191 is almost ready, I’m testing.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>>> Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>
>>>>> Hi guys,
>>>>>
>>>>> I’ve updated my local branch according to your comments:
>>>>>
>>>>> 1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
>>>>> 2. I’m keeping the smoke test profile
>>>>> 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
>>>>> 4. I’ve updated Jenkinsfile to use fast profile on PR
>>>>>
>>>>> I will create the PR soon.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>>>>>>
>>>>>> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>>>>>>
>>>>>> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>>>>>>
>>>>>> Thanks !
>>>>>> I will improve this way.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>>> Le 12 mars 2021 à 20:31, Clebert Suconic <clebert.suconic@gmail.com <ma...@gmail.com> <mailto:clebert.suconic@gmail.com <ma...@gmail.com>>> a écrit :
>>>>>>>
>>>>>>> You should instead have a fast profile, with a subset of the testsuite
>>>>>>> to run on every commit and branch for these cases. I looked on Jenkins
>>>>>>> and having many builds taking 3 Hours each won't really scale on the
>>>>>>> lab anyway. Failures will only make things worse there.
>>>>>>>
>>>>>>> The lab is usually not powerful for long running tests.
>>>>>>>
>>>>>>> And a full profile that should run as part of a full run. (say.. once
>>>>>>> a day instead of every commit), or any interval you chose.
>>>>>>>
>>>>>>> I don't think you should hide tests though.. as that is like pushing
>>>>>>> dirt under the rug.. (even if you say to enable it later... as in
>>>>>>> anything in life temporary solutions endup being definitive usually).
>>>>>>>
>>>>>>> As any System dealing with times and asynchronous flaky and races are
>>>>>>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>>>>>>> Rule where the test is retried. You could also add retries to tests in
>>>>>>> cases where it is acceptable... but be careful to not just hide bugs
>>>>>>> away in this case as well.
>>>>>>>
>>>>>>> If you are interested, on artemis, Look for usages on
>>>>>>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java> <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java>>
>>>>>>>
>>>>>>>
>>>>>>> You need to activate a profile in artemis for the retryRule to work.
>>>>>>>
>>>>>>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>
>>>>>>>> Yes agree. I’m launching new builds ;)
>>>>>>>>
>>>>>>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>> Just running it by itself on the command line and also in the IDE. The full
>>>>>>>>> build takes a while and if it's breaking with that then it's probably some
>>>>>>>>> other test that isn't cleaning up properly in between runs.
>>>>>>>>>
>>>>>>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>>>
>>>>>>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>>>>>>> build.
>>>>>>>>>>
>>>>>>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>>>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>>>>>>> box
>>>>>>>>>>> and it always passes.
>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>>>>>>> These
>>>>>>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>>>>>>> the
>>>>>>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>>>>>>> lead to
>>>>>>>>>>>> missing errors.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>>>>>>
>>>>>>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>>>>>>
>>>>>>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>>>>>>> Jenkins is happy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> JB
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>>>>>>> pipeline,
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>>>>>>> improve
>>>>>>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>>>>>>> PullRequests.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>>>>>>> containers)
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>>>>>>> "green
>>>>>>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Clebert Suconic
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Matt Pavlovich <ma...@gmail.com>.
Feels like we are in a transition period. I don’t see a per-PR unit test job being practical until the execution times come way down— and that is going to be significant engineering effort.
That being said, full build with full tests the day after a merged change seems like a reasonable schedule.
> On Mar 16, 2021, at 3:24 PM, Hossack, Etienne <eh...@amazon.com.INVALID> wrote:
>
> Hi all, just some thoughts to share here:
>
> I think ideally I would expect a “static” build to be a nightly build run every day - but I think given the frequency of contributions, weekly makes sense (and I know it takes about 30% of the day 😅).
> Maybe too, that runs a snapshot version of the dependency checker to fail the build on CVEs or new types of checks from that tool.
>
> And then for contributions, the PR can run the lightweight profile, but then master could run the full profile on merge?
>
> Does that make sense?
> In summary I think it’s me expressing agreement for a static build, but also suggesting a full build be run on contributions in case there are multiple merges in a week, or say right after the build is run, and increasing the time-to-discovery of errors.
>
> Cheers,
> Étienne Hossack
> Software Development Engineer, Amazon MQ
> email: ehossack@amazon.com <ma...@amazon.com>
> phone: +1-778-945-8287
>
>
>
>> On Mar 15, 2021, at 10:05 PM, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>
>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>>
>>
>>
>> Hi guys,
>>
>> I created https://github.com/apache/activemq/pull/622 <https://github.com/apache/activemq/pull/622> PR about this and you can see that Jenkins is happy now. The full build took about 120mn (2h) on Jenkins.
>>
>> Basically what I did in the PR:
>> - remove activemq-unit-tests and itests (Karaf, Spring3) from the default reactor
>> - introduce full.test profile that build all modules including unit tests and itests
>>
>> The full.test profile is not use in Jenkinsfile, meaning that the PR executes all tests but not activemq-unit-tests modules neither itests. I think it’s acceptable for PR (and it already takes 2 hours ;)).
>> I would like to introduce a "static" build on ci-builds.apache.org <http://ci-builds.apache.org/> (not via Jenkinsfile) executed every week and doing a full build (including full.test profile).
>>
>> Thoughts ?
>>
>> Regards
>> JB
>>
>>> Le 15 mars 2021 à 08:20, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>
>>> Hi guys,
>>>
>>> I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
>>>
>>> AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
>>> AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
>>> AMQ-8188: AMQ5266SingleDestTest is failing
>>>
>>> There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593 <https://github.com/apache/activemq/pull/593>).
>>>
>>> I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
>>>
>>> I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
>>>
>>> AMQ-8191 is almost ready, I’m testing.
>>>
>>> Regards
>>> JB
>>>
>>>> Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>>
>>>> Hi guys,
>>>>
>>>> I’ve updated my local branch according to your comments:
>>>>
>>>> 1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
>>>> 2. I’m keeping the smoke test profile
>>>> 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
>>>> 4. I’ve updated Jenkinsfile to use fast profile on PR
>>>>
>>>> I will create the PR soon.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>>> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>>>
>>>>> Hi,
>>>>>
>>>>> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>>>>>
>>>>> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>>>>>
>>>>> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>>>>>
>>>>> Thanks !
>>>>> I will improve this way.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> Le 12 mars 2021 à 20:31, Clebert Suconic <clebert.suconic@gmail.com <ma...@gmail.com>> a écrit :
>>>>>>
>>>>>> You should instead have a fast profile, with a subset of the testsuite
>>>>>> to run on every commit and branch for these cases. I looked on Jenkins
>>>>>> and having many builds taking 3 Hours each won't really scale on the
>>>>>> lab anyway. Failures will only make things worse there.
>>>>>>
>>>>>> The lab is usually not powerful for long running tests.
>>>>>>
>>>>>> And a full profile that should run as part of a full run. (say.. once
>>>>>> a day instead of every commit), or any interval you chose.
>>>>>>
>>>>>> I don't think you should hide tests though.. as that is like pushing
>>>>>> dirt under the rug.. (even if you say to enable it later... as in
>>>>>> anything in life temporary solutions endup being definitive usually).
>>>>>>
>>>>>> As any System dealing with times and asynchronous flaky and races are
>>>>>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>>>>>> Rule where the test is retried. You could also add retries to tests in
>>>>>> cases where it is acceptable... but be careful to not just hide bugs
>>>>>> away in this case as well.
>>>>>>
>>>>>> If you are interested, on artemis, Look for usages on
>>>>>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java <https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java>
>>>>>>
>>>>>>
>>>>>> You need to activate a profile in artemis for the retryRule to work.
>>>>>>
>>>>>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>
>>>>>>> Yes agree. I’m launching new builds ;)
>>>>>>>
>>>>>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>>>>>
>>>>>>>> Just running it by itself on the command line and also in the IDE. The full
>>>>>>>> build takes a while and if it's breaking with that then it's probably some
>>>>>>>> other test that isn't cleaning up properly in between runs.
>>>>>>>>
>>>>>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>>>
>>>>>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>>>>>> build.
>>>>>>>>>
>>>>>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>>>>>> box
>>>>>>>>>> and it always passes.
>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>>>>>> These
>>>>>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>>>>>> the
>>>>>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>>>>>> lead to
>>>>>>>>>>> missing errors.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>>>>>
>>>>>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>>>>>
>>>>>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>>>>>> Jenkins is happy.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> JB
>>>>>>>>>>>>
>>>>>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>>>>>> pipeline,
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>>>>>> improve
>>>>>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>>>>>> PullRequests.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>>>>>> containers)
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>>>>>> "green
>>>>>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Clebert Suconic
>>>>>
>>>>
>>>
>>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by "Hossack, Etienne" <eh...@amazon.com.INVALID>.
Hi all, just some thoughts to share here:
I think ideally I would expect a “static” build to be a nightly build run every day - but I think given the frequency of contributions, weekly makes sense (and I know it takes about 30% of the day 😅).
Maybe too, that runs a snapshot version of the dependency checker to fail the build on CVEs or new types of checks from that tool.
And then for contributions, the PR can run the lightweight profile, but then master could run the full profile on merge?
Does that make sense?
In summary I think it’s me expressing agreement for a static build, but also suggesting a full build be run on contributions in case there are multiple merges in a week, or say right after the build is run, and increasing the time-to-discovery of errors.
Cheers,
Étienne Hossack
Software Development Engineer, Amazon MQ
email: ehossack@amazon.com<ma...@amazon.com>
phone: +1-778-945-8287
[cid:0CFB4B72-BC66-42A8-953C-779D7FAE3DC0@amazon.com]
On Mar 15, 2021, at 10:05 PM, Jean-Baptiste Onofre <jb...@nanthrax.net>> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi guys,
I created https://github.com/apache/activemq/pull/622 PR about this and you can see that Jenkins is happy now. The full build took about 120mn (2h) on Jenkins.
Basically what I did in the PR:
- remove activemq-unit-tests and itests (Karaf, Spring3) from the default reactor
- introduce full.test profile that build all modules including unit tests and itests
The full.test profile is not use in Jenkinsfile, meaning that the PR executes all tests but not activemq-unit-tests modules neither itests. I think it’s acceptable for PR (and it already takes 2 hours ;)).
I would like to introduce a "static" build on ci-builds.apache.org<http://ci-builds.apache.org> (not via Jenkinsfile) executed every week and doing a full build (including full.test profile).
Thoughts ?
Regards
JB
Le 15 mars 2021 à 08:20, Jean-Baptiste Onofre <jb...@nanthrax.net>> a écrit :
Hi guys,
I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
AMQ-8188: AMQ5266SingleDestTest is failing
There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593).
I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
AMQ-8191 is almost ready, I’m testing.
Regards
JB
Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb...@nanthrax.net>> a écrit :
Hi guys,
I’ve updated my local branch according to your comments:
1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
2. I’m keeping the smoke test profile
3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
4. I’ve updated Jenkinsfile to use fast profile on PR
I will create the PR soon.
Regards
JB
Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb...@nanthrax.net>> a écrit :
Hi,
We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
Thanks !
I will improve this way.
Regards
JB
Le 12 mars 2021 à 20:31, Clebert Suconic <cl...@gmail.com>> a écrit :
You should instead have a fast profile, with a subset of the testsuite
to run on every commit and branch for these cases. I looked on Jenkins
and having many builds taking 3 Hours each won't really scale on the
lab anyway. Failures will only make things worse there.
The lab is usually not powerful for long running tests.
And a full profile that should run as part of a full run. (say.. once
a day instead of every commit), or any interval you chose.
I don't think you should hide tests though.. as that is like pushing
dirt under the rug.. (even if you say to enable it later... as in
anything in life temporary solutions endup being definitive usually).
As any System dealing with times and asynchronous flaky and races are
part of the day. One thing I did in ActiveMQ Artemis was to write a
Rule where the test is retried. You could also add retries to tests in
cases where it is acceptable... but be careful to not just hide bugs
away in this case as well.
If you are interested, on artemis, Look for usages on
https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java
You need to activate a profile in artemis for the retryRule to work.
On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
Yes agree. I’m launching new builds ;)
Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
Just running it by itself on the command line and also in the IDE. The full
build takes a while and if it's breaking with that then it's probably some
other test that isn't cleaning up properly in between runs.
On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
Did you try in a full build or the test individually ? I’m running a new
build.
Le 12 mars 2021 à 19:38, Christopher Shannon <
christopher.l.shannon@gmail.com> a écrit :
I've been running the DurableSyncNetworkBridgeTest several times on my
box
and it always passes.
On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
Ideally it would be better to fix tests than to simply exclude them.
These
tests were added for a reason I would presume (I know I had worked on
the
durable sync stuff in the past) so randomly turning off tests could
lead to
missing errors.
On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
wrote:
I’m adding these tests to be fixed/improved:
FailoverDurableSubTransactionTest.testFailoverCommitListener
DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
Let me create the Jira and create a PR to exclude the tests and verify
Jenkins is happy.
Regards
JB
Le 12 mars 2021 à 16:14, Jonathan Gallimore <
jonathan.gallimore@gmail.com> a écrit :
I'm +1 on the actions :).
Jon
On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
wrote:
Sure, thanks for the help !
Just waiting for some feedback before starting the "actions" ;)
Regards
JB
Le 12 mars 2021 à 14:29, Jonathan Gallimore <
jonathan.gallimore@gmail.com> a écrit :
I ran into this test failing yesterday:
activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
- I'd be happy to try and contribute a fix. Would you like to assign
the
JIRA to me?
Jon
On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
jb@nanthrax.net>
wrote:
Hi guys,
Now that we have Jenkinsfile in our repo, and we use Jenkins
pipeline,
we
dramatically improved our build: the build is executed for each
PullRequests or commit on the main branch.
However, we have lot of failing tests, causing quite systematically
the
build failing on ci-builds.apache.org.
We really need to have a clean, accurate and stable build: it will
improve
the issue detection and simplify the review, especially for
PullRequests.
I ran several builds on my machine (with different docker
containers)
and
I already identified some failing/flaky tests:
-
activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
is not a big deal as I have a PR removing leveled completely
-
activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
Chris did an improvement, but I still have some flakiness here.
-
activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
I propose the following action plan:
1. Create the Jira for each failing/flaky tests
2. Exclude the tests (in surefire plugin configuration) to have a
"green
light" on Jenkins.
3. For each Jira, we work on a PullRequest, to be sure that Jenkins
is
still "happy".
Anyone willing to help on (3) is welcome !
If there’s no objection, I will start with (1) and (2).
Thanks,
Regards
JB
--
Clebert Suconic
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi guys,
I created https://github.com/apache/activemq/pull/622 PR about this and you can see that Jenkins is happy now. The full build took about 120mn (2h) on Jenkins.
Basically what I did in the PR:
- remove activemq-unit-tests and itests (Karaf, Spring3) from the default reactor
- introduce full.test profile that build all modules including unit tests and itests
The full.test profile is not use in Jenkinsfile, meaning that the PR executes all tests but not activemq-unit-tests modules neither itests. I think it’s acceptable for PR (and it already takes 2 hours ;)).
I would like to introduce a "static" build on ci-builds.apache.org (not via Jenkinsfile) executed every week and doing a full build (including full.test profile).
Thoughts ?
Regards
JB
> Le 15 mars 2021 à 08:20, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>
> Hi guys,
>
> I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
>
> AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
> AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
> AMQ-8188: AMQ5266SingleDestTest is failing
>
> There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593).
>
> I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
>
> I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
>
> AMQ-8191 is almost ready, I’m testing.
>
> Regards
> JB
>
>> Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>>
>> Hi guys,
>>
>> I’ve updated my local branch according to your comments:
>>
>> 1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
>> 2. I’m keeping the smoke test profile
>> 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
>> 4. I’ve updated Jenkinsfile to use fast profile on PR
>>
>> I will create the PR soon.
>>
>> Regards
>> JB
>>
>>> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>>>
>>> Hi,
>>>
>>> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>>>
>>> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>>>
>>> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>>>
>>> Thanks !
>>> I will improve this way.
>>>
>>> Regards
>>> JB
>>>
>>>> Le 12 mars 2021 à 20:31, Clebert Suconic <cl...@gmail.com> a écrit :
>>>>
>>>> You should instead have a fast profile, with a subset of the testsuite
>>>> to run on every commit and branch for these cases. I looked on Jenkins
>>>> and having many builds taking 3 Hours each won't really scale on the
>>>> lab anyway. Failures will only make things worse there.
>>>>
>>>> The lab is usually not powerful for long running tests.
>>>>
>>>> And a full profile that should run as part of a full run. (say.. once
>>>> a day instead of every commit), or any interval you chose.
>>>>
>>>> I don't think you should hide tests though.. as that is like pushing
>>>> dirt under the rug.. (even if you say to enable it later... as in
>>>> anything in life temporary solutions endup being definitive usually).
>>>>
>>>> As any System dealing with times and asynchronous flaky and races are
>>>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>>>> Rule where the test is retried. You could also add retries to tests in
>>>> cases where it is acceptable... but be careful to not just hide bugs
>>>> away in this case as well.
>>>>
>>>> If you are interested, on artemis, Look for usages on
>>>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java
>>>>
>>>>
>>>> You need to activate a profile in artemis for the retryRule to work.
>>>>
>>>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>
>>>>> Yes agree. I’m launching new builds ;)
>>>>>
>>>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>>>
>>>>>> Just running it by itself on the command line and also in the IDE. The full
>>>>>> build takes a while and if it's breaking with that then it's probably some
>>>>>> other test that isn't cleaning up properly in between runs.
>>>>>>
>>>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>>
>>>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>>>> build.
>>>>>>>
>>>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>>>
>>>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>>>> box
>>>>>>>> and it always passes.
>>>>>>>>
>>>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>>>> These
>>>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>>>> the
>>>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>>>> lead to
>>>>>>>>> missing errors.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>>>
>>>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>>>
>>>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>>>> Jenkins is happy.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> JB
>>>>>>>>>>
>>>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>>>
>>>>>>>>>>> Jon
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>>>
>>>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> JB
>>>>>>>>>>>>
>>>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>>>> the
>>>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Jon
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>>>> pipeline,
>>>>>>>>>>>> we
>>>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>>>> the
>>>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>>>> improve
>>>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>>>> PullRequests.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>>>> containers)
>>>>>>>>>>>> and
>>>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>>>> -
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>>>> "green
>>>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>>>> is
>>>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> JB
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Clebert Suconic
>>>
>>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi guys,
I have create the following Jira with the tests I found "flaky" (in a full build, not necessary single execution, it can also depends of the machine, that’s why I tested with several docker setup in terms of CPU and memory):
AMQ-8190: DuplexAdvisoryRaceTest is failing (Jonathan said he gonna take a look)
AMQ-8189: CachedLDAPAuthorizationModuleTest is failing
AMQ-8188: AMQ5266SingleDestTest is failing
There’s a test failure in leveldb module, but it’s not a big deal as I have the PR ready to remove leveldb (https://github.com/apache/activemq/pull/593).
I’m also retesting StompNIOSSLTest, it seems way more stable thanks to Chris
I also created AMQ-8191 (linked with previous Jira) about cleanup on the profiles, fast.test profile introduction and usage on Jenkins, and exclude the failing tests waiting to be fixed (and reinclude them at that time).
AMQ-8191 is almost ready, I’m testing.
Regards
JB
> Le 14 mars 2021 à 06:04, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>
> Hi guys,
>
> I’ve updated my local branch according to your comments:
>
> 1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
> 2. I’m keeping the smoke test profile
> 3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
> 4. I’ve updated Jenkinsfile to use fast profile on PR
>
> I will create the PR soon.
>
> Regards
> JB
>
>> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>>
>> Hi,
>>
>> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>>
>> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>>
>> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>>
>> Thanks !
>> I will improve this way.
>>
>> Regards
>> JB
>>
>>> Le 12 mars 2021 à 20:31, Clebert Suconic <cl...@gmail.com> a écrit :
>>>
>>> You should instead have a fast profile, with a subset of the testsuite
>>> to run on every commit and branch for these cases. I looked on Jenkins
>>> and having many builds taking 3 Hours each won't really scale on the
>>> lab anyway. Failures will only make things worse there.
>>>
>>> The lab is usually not powerful for long running tests.
>>>
>>> And a full profile that should run as part of a full run. (say.. once
>>> a day instead of every commit), or any interval you chose.
>>>
>>> I don't think you should hide tests though.. as that is like pushing
>>> dirt under the rug.. (even if you say to enable it later... as in
>>> anything in life temporary solutions endup being definitive usually).
>>>
>>> As any System dealing with times and asynchronous flaky and races are
>>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>>> Rule where the test is retried. You could also add retries to tests in
>>> cases where it is acceptable... but be careful to not just hide bugs
>>> away in this case as well.
>>>
>>> If you are interested, on artemis, Look for usages on
>>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java
>>>
>>>
>>> You need to activate a profile in artemis for the retryRule to work.
>>>
>>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Yes agree. I’m launching new builds ;)
>>>>
>>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>>
>>>>> Just running it by itself on the command line and also in the IDE. The full
>>>>> build takes a while and if it's breaking with that then it's probably some
>>>>> other test that isn't cleaning up properly in between runs.
>>>>>
>>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>>
>>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>>> build.
>>>>>>
>>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>>
>>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>>> box
>>>>>>> and it always passes.
>>>>>>>
>>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>>> These
>>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>>> the
>>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>>> lead to
>>>>>>>> missing errors.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>>
>>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>>
>>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>>> Jenkins is happy.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> JB
>>>>>>>>>
>>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>>
>>>>>>>>>> Jon
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>>
>>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> JB
>>>>>>>>>>>
>>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>>> the
>>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>>
>>>>>>>>>>>> Jon
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>>> pipeline,
>>>>>>>>>>> we
>>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>>
>>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>>> the
>>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>>> improve
>>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>>> PullRequests.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>>> containers)
>>>>>>>>>>> and
>>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>>
>>>>>>>>>>>>> -
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>>> -
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>>> -
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>>> "green
>>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>>> is
>>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>>
>>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>>
>>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> JB
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi guys,
I’ve updated my local branch according to your comments:
1. I’ve cleanup the profiles and introduce/rename a fast profile that executes all unit tests in modules but exclude the activemq-unit-tests and karaf-itests.
2. I’m keeping the smoke test profile
3. I’ve created a tobefixed profile that include all flaky tests I’ve identified
4. I’ve updated Jenkinsfile to use fast profile on PR
I will create the PR soon.
Regards
JB
> Le 13 mars 2021 à 06:05, Jean-Baptiste Onofre <jb...@nanthrax.net> a écrit :
>
> Hi,
>
> We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
>
> For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
>
> About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
>
> Thanks !
> I will improve this way.
>
> Regards
> JB
>
>> Le 12 mars 2021 à 20:31, Clebert Suconic <cl...@gmail.com> a écrit :
>>
>> You should instead have a fast profile, with a subset of the testsuite
>> to run on every commit and branch for these cases. I looked on Jenkins
>> and having many builds taking 3 Hours each won't really scale on the
>> lab anyway. Failures will only make things worse there.
>>
>> The lab is usually not powerful for long running tests.
>>
>> And a full profile that should run as part of a full run. (say.. once
>> a day instead of every commit), or any interval you chose.
>>
>> I don't think you should hide tests though.. as that is like pushing
>> dirt under the rug.. (even if you say to enable it later... as in
>> anything in life temporary solutions endup being definitive usually).
>>
>> As any System dealing with times and asynchronous flaky and races are
>> part of the day. One thing I did in ActiveMQ Artemis was to write a
>> Rule where the test is retried. You could also add retries to tests in
>> cases where it is acceptable... but be careful to not just hide bugs
>> away in this case as well.
>>
>> If you are interested, on artemis, Look for usages on
>> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java
>>
>>
>> You need to activate a profile in artemis for the retryRule to work.
>>
>> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>
>>> Yes agree. I’m launching new builds ;)
>>>
>>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>>
>>>> Just running it by itself on the command line and also in the IDE. The full
>>>> build takes a while and if it's breaking with that then it's probably some
>>>> other test that isn't cleaning up properly in between runs.
>>>>
>>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>>
>>>>> Did you try in a full build or the test individually ? I’m running a new
>>>>> build.
>>>>>
>>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>>
>>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>>> box
>>>>>> and it always passes.
>>>>>>
>>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>>
>>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>>> These
>>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>>> the
>>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>>> lead to
>>>>>>> missing errors.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>>
>>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>>
>>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>>> Jenkins is happy.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> JB
>>>>>>>>
>>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>> I'm +1 on the actions :).
>>>>>>>>>
>>>>>>>>> Jon
>>>>>>>>>
>>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>>
>>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> JB
>>>>>>>>>>
>>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>>> the
>>>>>>>>>>> JIRA to me?
>>>>>>>>>>>
>>>>>>>>>>> Jon
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>>> jb@nanthrax.net>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi guys,
>>>>>>>>>>>>
>>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>>> pipeline,
>>>>>>>>>> we
>>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>>
>>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>>> the
>>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>>
>>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>>> improve
>>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>>> PullRequests.
>>>>>>>>>>>>
>>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>>> containers)
>>>>>>>>>> and
>>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>>
>>>>>>>>>>>> -
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>>> -
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>>> -
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>>
>>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>>> "green
>>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>>> is
>>>>>>>>>>>> still "happy".
>>>>>>>>>>>>
>>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>>
>>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Regards
>>>>>>>>>>>> JB
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>>
>>>
>>
>>
>> --
>> Clebert Suconic
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Hi,
We already have "fast" profile, and it’s good idea to use this profile on Jenkins by default and move some tests here.
For instance, I don’t think it’s require to launch all activemq-unit-test by default but I would keep the tests in each module (they are fast and doesn’t need whole broker infra).
About RetryRule, I did that in Karaf as well, let me see if it helps for ActiveMQ.
Thanks !
I will improve this way.
Regards
JB
> Le 12 mars 2021 à 20:31, Clebert Suconic <cl...@gmail.com> a écrit :
>
> You should instead have a fast profile, with a subset of the testsuite
> to run on every commit and branch for these cases. I looked on Jenkins
> and having many builds taking 3 Hours each won't really scale on the
> lab anyway. Failures will only make things worse there.
>
> The lab is usually not powerful for long running tests.
>
> And a full profile that should run as part of a full run. (say.. once
> a day instead of every commit), or any interval you chose.
>
> I don't think you should hide tests though.. as that is like pushing
> dirt under the rug.. (even if you say to enable it later... as in
> anything in life temporary solutions endup being definitive usually).
>
> As any System dealing with times and asynchronous flaky and races are
> part of the day. One thing I did in ActiveMQ Artemis was to write a
> Rule where the test is retried. You could also add retries to tests in
> cases where it is acceptable... but be careful to not just hide bugs
> away in this case as well.
>
> If you are interested, on artemis, Look for usages on
> https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java
>
>
> You need to activate a profile in artemis for the retryRule to work.
>
> On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>>
>> Yes agree. I’m launching new builds ;)
>>
>>> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>>>
>>> Just running it by itself on the command line and also in the IDE. The full
>>> build takes a while and if it's breaking with that then it's probably some
>>> other test that isn't cleaning up properly in between runs.
>>>
>>>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Did you try in a full build or the test individually ? I’m running a new
>>>> build.
>>>>
>>>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>>>> christopher.l.shannon@gmail.com> a écrit :
>>>>>
>>>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>>>> box
>>>>> and it always passes.
>>>>>
>>>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>>>> christopher.l.shannon@gmail.com> wrote:
>>>>>>
>>>>>> Ideally it would be better to fix tests than to simply exclude them.
>>>> These
>>>>>> tests were added for a reason I would presume (I know I had worked on
>>>> the
>>>>>> durable sync stuff in the past) so randomly turning off tests could
>>>> lead to
>>>>>> missing errors.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>>> I’m adding these tests to be fixed/improved:
>>>>>>>
>>>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>>>
>>>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>>>> Jenkins is happy.
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>
>>>>>>>> I'm +1 on the actions :).
>>>>>>>>
>>>>>>>> Jon
>>>>>>>>
>>>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Sure, thanks for the help !
>>>>>>>>>
>>>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> JB
>>>>>>>>>
>>>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>>>> the
>>>>>>>>>> JIRA to me?
>>>>>>>>>>
>>>>>>>>>> Jon
>>>>>>>>>>
>>>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>>>> jb@nanthrax.net>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>
>>>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>>>> pipeline,
>>>>>>>>> we
>>>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>>>
>>>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>>>> the
>>>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>>>
>>>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>>>> improve
>>>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>>>> PullRequests.
>>>>>>>>>>>
>>>>>>>>>>> I ran several builds on my machine (with different docker
>>>> containers)
>>>>>>>>> and
>>>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>>>
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>>>> -
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>>>
>>>>>>>>>>> I propose the following action plan:
>>>>>>>>>>>
>>>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>>>> "green
>>>>>>>>>>> light" on Jenkins.
>>>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>>>> is
>>>>>>>>>>> still "happy".
>>>>>>>>>>>
>>>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>>>
>>>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Regards
>>>>>>>>>>> JB
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>>
>
>
> --
> Clebert Suconic
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Clebert Suconic <cl...@gmail.com>.
You should instead have a fast profile, with a subset of the testsuite
to run on every commit and branch for these cases. I looked on Jenkins
and having many builds taking 3 Hours each won't really scale on the
lab anyway. Failures will only make things worse there.
The lab is usually not powerful for long running tests.
And a full profile that should run as part of a full run. (say.. once
a day instead of every commit), or any interval you chose.
I don't think you should hide tests though.. as that is like pushing
dirt under the rug.. (even if you say to enable it later... as in
anything in life temporary solutions endup being definitive usually).
As any System dealing with times and asynchronous flaky and races are
part of the day. One thing I did in ActiveMQ Artemis was to write a
Rule where the test is retried. You could also add retries to tests in
cases where it is acceptable... but be careful to not just hide bugs
away in this case as well.
If you are interested, on artemis, Look for usages on
https://github.com/apache/activemq-artemis/blob/master/artemis-commons/src/test/java/org/apache/activemq/artemis/utils/RetryRule.java
You need to activate a profile in artemis for the retryRule to work.
On Fri, Mar 12, 2021 at 1:56 PM JB Onofré <jb...@nanthrax.net> wrote:
>
> Yes agree. I’m launching new builds ;)
>
> > Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
> >
> > Just running it by itself on the command line and also in the IDE. The full
> > build takes a while and if it's breaking with that then it's probably some
> > other test that isn't cleaning up properly in between runs.
> >
> >> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
> >>
> >> Did you try in a full build or the test individually ? I’m running a new
> >> build.
> >>
> >>> Le 12 mars 2021 à 19:38, Christopher Shannon <
> >> christopher.l.shannon@gmail.com> a écrit :
> >>>
> >>> I've been running the DurableSyncNetworkBridgeTest several times on my
> >> box
> >>> and it always passes.
> >>>
> >>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
> >>>> christopher.l.shannon@gmail.com> wrote:
> >>>>
> >>>> Ideally it would be better to fix tests than to simply exclude them.
> >> These
> >>>> tests were added for a reason I would presume (I know I had worked on
> >> the
> >>>> durable sync stuff in the past) so randomly turning off tests could
> >> lead to
> >>>> missing errors.
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
> >>>> wrote:
> >>>>
> >>>>> I’m adding these tests to be fixed/improved:
> >>>>>
> >>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
> >>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
> >>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
> >>>>>
> >>>>> Let me create the Jira and create a PR to exclude the tests and verify
> >>>>> Jenkins is happy.
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
> >>>>> jonathan.gallimore@gmail.com> a écrit :
> >>>>>>
> >>>>>> I'm +1 on the actions :).
> >>>>>>
> >>>>>> Jon
> >>>>>>
> >>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
> >>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Sure, thanks for the help !
> >>>>>>>
> >>>>>>> Just waiting for some feedback before starting the "actions" ;)
> >>>>>>>
> >>>>>>> Regards
> >>>>>>> JB
> >>>>>>>
> >>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
> >>>>>>> jonathan.gallimore@gmail.com> a écrit :
> >>>>>>>>
> >>>>>>>> I ran into this test failing yesterday:
> >>>>>>>>
> >>>>>>>
> >>>>>
> >> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> >>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
> >>>>> the
> >>>>>>>> JIRA to me?
> >>>>>>>>
> >>>>>>>> Jon
> >>>>>>>>
> >>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
> >>>>> jb@nanthrax.net>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi guys,
> >>>>>>>>>
> >>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
> >>>>> pipeline,
> >>>>>>> we
> >>>>>>>>> dramatically improved our build: the build is executed for each
> >>>>>>>>> PullRequests or commit on the main branch.
> >>>>>>>>>
> >>>>>>>>> However, we have lot of failing tests, causing quite systematically
> >>>>> the
> >>>>>>>>> build failing on ci-builds.apache.org.
> >>>>>>>>>
> >>>>>>>>> We really need to have a clean, accurate and stable build: it will
> >>>>>>> improve
> >>>>>>>>> the issue detection and simplify the review, especially for
> >>>>>>> PullRequests.
> >>>>>>>>>
> >>>>>>>>> I ran several builds on my machine (with different docker
> >> containers)
> >>>>>>> and
> >>>>>>>>> I already identified some failing/flaky tests:
> >>>>>>>>>
> >>>>>>>>> -
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
> >>>>>>>>> is not a big deal as I have a PR removing leveled completely
> >>>>>>>>> -
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
> >>>>>>>>> Chris did an improvement, but I still have some flakiness here.
> >>>>>>>>> -
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> >>>>>>>>>
> >>>>>>>>> I propose the following action plan:
> >>>>>>>>>
> >>>>>>>>> 1. Create the Jira for each failing/flaky tests
> >>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
> >>>>> "green
> >>>>>>>>> light" on Jenkins.
> >>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
> >>>>> is
> >>>>>>>>> still "happy".
> >>>>>>>>>
> >>>>>>>>> Anyone willing to help on (3) is welcome !
> >>>>>>>>>
> >>>>>>>>> If there’s no objection, I will start with (1) and (2).
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Regards
> >>>>>>>>> JB
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>
> >>
>
--
Clebert Suconic
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by JB Onofré <jb...@nanthrax.net>.
Yes agree. I’m launching new builds ;)
> Le 12 mars 2021 à 19:51, Christopher Shannon <ch...@gmail.com> a écrit :
>
> Just running it by itself on the command line and also in the IDE. The full
> build takes a while and if it's breaking with that then it's probably some
> other test that isn't cleaning up properly in between runs.
>
>> On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
>>
>> Did you try in a full build or the test individually ? I’m running a new
>> build.
>>
>>> Le 12 mars 2021 à 19:38, Christopher Shannon <
>> christopher.l.shannon@gmail.com> a écrit :
>>>
>>> I've been running the DurableSyncNetworkBridgeTest several times on my
>> box
>>> and it always passes.
>>>
>>>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>>>> christopher.l.shannon@gmail.com> wrote:
>>>>
>>>> Ideally it would be better to fix tests than to simply exclude them.
>> These
>>>> tests were added for a reason I would presume (I know I had worked on
>> the
>>>> durable sync stuff in the past) so randomly turning off tests could
>> lead to
>>>> missing errors.
>>>>
>>>>
>>>>
>>>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> I’m adding these tests to be fixed/improved:
>>>>>
>>>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>>>
>>>>> Let me create the Jira and create a PR to exclude the tests and verify
>>>>> Jenkins is happy.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>
>>>>>> I'm +1 on the actions :).
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Sure, thanks for the help !
>>>>>>>
>>>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>>>
>>>>>>>> I ran into this test failing yesterday:
>>>>>>>>
>>>>>>>
>>>>>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>>>> the
>>>>>>>> JIRA to me?
>>>>>>>>
>>>>>>>> Jon
>>>>>>>>
>>>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>>>> jb@nanthrax.net>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi guys,
>>>>>>>>>
>>>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>>>> pipeline,
>>>>>>> we
>>>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>>>> PullRequests or commit on the main branch.
>>>>>>>>>
>>>>>>>>> However, we have lot of failing tests, causing quite systematically
>>>>> the
>>>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>>>
>>>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>>>> improve
>>>>>>>>> the issue detection and simplify the review, especially for
>>>>>>> PullRequests.
>>>>>>>>>
>>>>>>>>> I ran several builds on my machine (with different docker
>> containers)
>>>>>>> and
>>>>>>>>> I already identified some failing/flaky tests:
>>>>>>>>>
>>>>>>>>> -
>>>>>>>>>
>>>>>>>
>>>>>
>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>>>> -
>>>>>>>>>
>>>>>>>
>>>>>
>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>>>> -
>>>>>>>>>
>>>>>>>
>>>>>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>>>
>>>>>>>>> I propose the following action plan:
>>>>>>>>>
>>>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>>>> "green
>>>>>>>>> light" on Jenkins.
>>>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>>>> is
>>>>>>>>> still "happy".
>>>>>>>>>
>>>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>>>
>>>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Regards
>>>>>>>>> JB
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>
>>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Christopher Shannon <ch...@gmail.com>.
Just running it by itself on the command line and also in the IDE. The full
build takes a while and if it's breaking with that then it's probably some
other test that isn't cleaning up properly in between runs.
On Fri, Mar 12, 2021 at 1:47 PM JB Onofré <jb...@nanthrax.net> wrote:
> Did you try in a full build or the test individually ? I’m running a new
> build.
>
> > Le 12 mars 2021 à 19:38, Christopher Shannon <
> christopher.l.shannon@gmail.com> a écrit :
> >
> > I've been running the DurableSyncNetworkBridgeTest several times on my
> box
> > and it always passes.
> >
> >> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
> >> christopher.l.shannon@gmail.com> wrote:
> >>
> >> Ideally it would be better to fix tests than to simply exclude them.
> These
> >> tests were added for a reason I would presume (I know I had worked on
> the
> >> durable sync stuff in the past) so randomly turning off tests could
> lead to
> >> missing errors.
> >>
> >>
> >>
> >> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
> >> wrote:
> >>
> >>> I’m adding these tests to be fixed/improved:
> >>>
> >>> FailoverDurableSubTransactionTest.testFailoverCommitListener
> >>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
> >>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
> >>>
> >>> Let me create the Jira and create a PR to exclude the tests and verify
> >>> Jenkins is happy.
> >>>
> >>> Regards
> >>> JB
> >>>
> >>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
> >>> jonathan.gallimore@gmail.com> a écrit :
> >>>>
> >>>> I'm +1 on the actions :).
> >>>>
> >>>> Jon
> >>>>
> >>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb@nanthrax.net
> >
> >>>> wrote:
> >>>>
> >>>>> Sure, thanks for the help !
> >>>>>
> >>>>> Just waiting for some feedback before starting the "actions" ;)
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
> >>>>> jonathan.gallimore@gmail.com> a écrit :
> >>>>>>
> >>>>>> I ran into this test failing yesterday:
> >>>>>>
> >>>>>
> >>>
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> >>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
> >>> the
> >>>>>> JIRA to me?
> >>>>>>
> >>>>>> Jon
> >>>>>>
> >>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
> >>> jb@nanthrax.net>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hi guys,
> >>>>>>>
> >>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
> >>> pipeline,
> >>>>> we
> >>>>>>> dramatically improved our build: the build is executed for each
> >>>>>>> PullRequests or commit on the main branch.
> >>>>>>>
> >>>>>>> However, we have lot of failing tests, causing quite systematically
> >>> the
> >>>>>>> build failing on ci-builds.apache.org.
> >>>>>>>
> >>>>>>> We really need to have a clean, accurate and stable build: it will
> >>>>> improve
> >>>>>>> the issue detection and simplify the review, especially for
> >>>>> PullRequests.
> >>>>>>>
> >>>>>>> I ran several builds on my machine (with different docker
> containers)
> >>>>> and
> >>>>>>> I already identified some failing/flaky tests:
> >>>>>>>
> >>>>>>> -
> >>>>>>>
> >>>>>
> >>>
> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
> >>>>>>> is not a big deal as I have a PR removing leveled completely
> >>>>>>> -
> >>>>>>>
> >>>>>
> >>>
> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
> >>>>>>> Chris did an improvement, but I still have some flakiness here.
> >>>>>>> -
> >>>>>>>
> >>>>>
> >>>
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> >>>>>>>
> >>>>>>> I propose the following action plan:
> >>>>>>>
> >>>>>>> 1. Create the Jira for each failing/flaky tests
> >>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
> >>> "green
> >>>>>>> light" on Jenkins.
> >>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
> >>> is
> >>>>>>> still "happy".
> >>>>>>>
> >>>>>>> Anyone willing to help on (3) is welcome !
> >>>>>>>
> >>>>>>> If there’s no objection, I will start with (1) and (2).
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Regards
> >>>>>>> JB
> >>>>>
> >>>>>
> >>>
> >>>
>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by JB Onofré <jb...@nanthrax.net>.
Did you try in a full build or the test individually ? I’m running a new build.
> Le 12 mars 2021 à 19:38, Christopher Shannon <ch...@gmail.com> a écrit :
>
> I've been running the DurableSyncNetworkBridgeTest several times on my box
> and it always passes.
>
>> On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
>> christopher.l.shannon@gmail.com> wrote:
>>
>> Ideally it would be better to fix tests than to simply exclude them. These
>> tests were added for a reason I would presume (I know I had worked on the
>> durable sync stuff in the past) so randomly turning off tests could lead to
>> missing errors.
>>
>>
>>
>> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>> wrote:
>>
>>> I’m adding these tests to be fixed/improved:
>>>
>>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>>
>>> Let me create the Jira and create a PR to exclude the tests and verify
>>> Jenkins is happy.
>>>
>>> Regards
>>> JB
>>>
>>>> Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>>> jonathan.gallimore@gmail.com> a écrit :
>>>>
>>>> I'm +1 on the actions :).
>>>>
>>>> Jon
>>>>
>>>> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> Sure, thanks for the help !
>>>>>
>>>>> Just waiting for some feedback before starting the "actions" ;)
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>>>>> jonathan.gallimore@gmail.com> a écrit :
>>>>>>
>>>>>> I ran into this test failing yesterday:
>>>>>>
>>>>>
>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>> - I'd be happy to try and contribute a fix. Would you like to assign
>>> the
>>>>>> JIRA to me?
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>>> jb@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>>> pipeline,
>>>>> we
>>>>>>> dramatically improved our build: the build is executed for each
>>>>>>> PullRequests or commit on the main branch.
>>>>>>>
>>>>>>> However, we have lot of failing tests, causing quite systematically
>>> the
>>>>>>> build failing on ci-builds.apache.org.
>>>>>>>
>>>>>>> We really need to have a clean, accurate and stable build: it will
>>>>> improve
>>>>>>> the issue detection and simplify the review, especially for
>>>>> PullRequests.
>>>>>>>
>>>>>>> I ran several builds on my machine (with different docker containers)
>>>>> and
>>>>>>> I already identified some failing/flaky tests:
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>
>>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>>>>> is not a big deal as I have a PR removing leveled completely
>>>>>>> -
>>>>>>>
>>>>>
>>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>>>>> Chris did an improvement, but I still have some flakiness here.
>>>>>>> -
>>>>>>>
>>>>>
>>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>>>>
>>>>>>> I propose the following action plan:
>>>>>>>
>>>>>>> 1. Create the Jira for each failing/flaky tests
>>>>>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>>> "green
>>>>>>> light" on Jenkins.
>>>>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>>> is
>>>>>>> still "happy".
>>>>>>>
>>>>>>> Anyone willing to help on (3) is welcome !
>>>>>>>
>>>>>>> If there’s no objection, I will start with (1) and (2).
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Regards
>>>>>>> JB
>>>>>
>>>>>
>>>
>>>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Christopher Shannon <ch...@gmail.com>.
I've been running the DurableSyncNetworkBridgeTest several times on my box
and it always passes.
On Fri, Mar 12, 2021 at 1:25 PM Christopher Shannon <
christopher.l.shannon@gmail.com> wrote:
> Ideally it would be better to fix tests than to simply exclude them. These
> tests were added for a reason I would presume (I know I had worked on the
> durable sync stuff in the past) so randomly turning off tests could lead to
> missing errors.
>
>
>
> On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
> wrote:
>
>> I’m adding these tests to be fixed/improved:
>>
>> FailoverDurableSubTransactionTest.testFailoverCommitListener
>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
>> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>>
>> Let me create the Jira and create a PR to exclude the tests and verify
>> Jenkins is happy.
>>
>> Regards
>> JB
>>
>> > Le 12 mars 2021 à 16:14, Jonathan Gallimore <
>> jonathan.gallimore@gmail.com> a écrit :
>> >
>> > I'm +1 on the actions :).
>> >
>> > Jon
>> >
>> > On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>> > wrote:
>> >
>> >> Sure, thanks for the help !
>> >>
>> >> Just waiting for some feedback before starting the "actions" ;)
>> >>
>> >> Regards
>> >> JB
>> >>
>> >>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>> >> jonathan.gallimore@gmail.com> a écrit :
>> >>>
>> >>> I ran into this test failing yesterday:
>> >>>
>> >>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>> >>> - I'd be happy to try and contribute a fix. Would you like to assign
>> the
>> >>> JIRA to me?
>> >>>
>> >>> Jon
>> >>>
>> >>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <
>> jb@nanthrax.net>
>> >>> wrote:
>> >>>
>> >>>> Hi guys,
>> >>>>
>> >>>> Now that we have Jenkinsfile in our repo, and we use Jenkins
>> pipeline,
>> >> we
>> >>>> dramatically improved our build: the build is executed for each
>> >>>> PullRequests or commit on the main branch.
>> >>>>
>> >>>> However, we have lot of failing tests, causing quite systematically
>> the
>> >>>> build failing on ci-builds.apache.org.
>> >>>>
>> >>>> We really need to have a clean, accurate and stable build: it will
>> >> improve
>> >>>> the issue detection and simplify the review, especially for
>> >> PullRequests.
>> >>>>
>> >>>> I ran several builds on my machine (with different docker containers)
>> >> and
>> >>>> I already identified some failing/flaky tests:
>> >>>>
>> >>>> -
>> >>>>
>> >>
>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>> >>>> is not a big deal as I have a PR removing leveled completely
>> >>>> -
>> >>>>
>> >>
>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>> >>>> Chris did an improvement, but I still have some flakiness here.
>> >>>> -
>> >>>>
>> >>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>> >>>>
>> >>>> I propose the following action plan:
>> >>>>
>> >>>> 1. Create the Jira for each failing/flaky tests
>> >>>> 2. Exclude the tests (in surefire plugin configuration) to have a
>> "green
>> >>>> light" on Jenkins.
>> >>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins
>> is
>> >>>> still "happy".
>> >>>>
>> >>>> Anyone willing to help on (3) is welcome !
>> >>>>
>> >>>> If there’s no objection, I will start with (1) and (2).
>> >>>>
>> >>>> Thanks,
>> >>>> Regards
>> >>>> JB
>> >>
>> >>
>>
>>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Christopher Shannon <ch...@gmail.com>.
Ideally it would be better to fix tests than to simply exclude them. These
tests were added for a reason I would presume (I know I had worked on the
durable sync stuff in the past) so randomly turning off tests could lead to
missing errors.
On Fri, Mar 12, 2021 at 12:57 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
wrote:
> I’m adding these tests to be fixed/improved:
>
> FailoverDurableSubTransactionTest.testFailoverCommitListener
> DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
> DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
>
> Let me create the Jira and create a PR to exclude the tests and verify
> Jenkins is happy.
>
> Regards
> JB
>
> > Le 12 mars 2021 à 16:14, Jonathan Gallimore <
> jonathan.gallimore@gmail.com> a écrit :
> >
> > I'm +1 on the actions :).
> >
> > Jon
> >
> > On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
> > wrote:
> >
> >> Sure, thanks for the help !
> >>
> >> Just waiting for some feedback before starting the "actions" ;)
> >>
> >> Regards
> >> JB
> >>
> >>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
> >> jonathan.gallimore@gmail.com> a écrit :
> >>>
> >>> I ran into this test failing yesterday:
> >>>
> >>
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> >>> - I'd be happy to try and contribute a fix. Would you like to assign
> the
> >>> JIRA to me?
> >>>
> >>> Jon
> >>>
> >>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <jb@nanthrax.net
> >
> >>> wrote:
> >>>
> >>>> Hi guys,
> >>>>
> >>>> Now that we have Jenkinsfile in our repo, and we use Jenkins pipeline,
> >> we
> >>>> dramatically improved our build: the build is executed for each
> >>>> PullRequests or commit on the main branch.
> >>>>
> >>>> However, we have lot of failing tests, causing quite systematically
> the
> >>>> build failing on ci-builds.apache.org.
> >>>>
> >>>> We really need to have a clean, accurate and stable build: it will
> >> improve
> >>>> the issue detection and simplify the review, especially for
> >> PullRequests.
> >>>>
> >>>> I ran several builds on my machine (with different docker containers)
> >> and
> >>>> I already identified some failing/flaky tests:
> >>>>
> >>>> -
> >>>>
> >>
> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
> >>>> is not a big deal as I have a PR removing leveled completely
> >>>> -
> >>>>
> >>
> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
> >>>> Chris did an improvement, but I still have some flakiness here.
> >>>> -
> >>>>
> >>
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> >>>>
> >>>> I propose the following action plan:
> >>>>
> >>>> 1. Create the Jira for each failing/flaky tests
> >>>> 2. Exclude the tests (in surefire plugin configuration) to have a
> "green
> >>>> light" on Jenkins.
> >>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins is
> >>>> still "happy".
> >>>>
> >>>> Anyone willing to help on (3) is welcome !
> >>>>
> >>>> If there’s no objection, I will start with (1) and (2).
> >>>>
> >>>> Thanks,
> >>>> Regards
> >>>> JB
> >>
> >>
>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
I’m adding these tests to be fixed/improved:
FailoverDurableSubTransactionTest.testFailoverCommitListener
DurableSyncNetworkBridgeTest.testRemoveSubscriptionPropagate
DurableSyncNetworkBridgeTest.testRemoveSubscriptionWithBridgeOffline
Let me create the Jira and create a PR to exclude the tests and verify Jenkins is happy.
Regards
JB
> Le 12 mars 2021 à 16:14, Jonathan Gallimore <jo...@gmail.com> a écrit :
>
> I'm +1 on the actions :).
>
> Jon
>
> On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
> wrote:
>
>> Sure, thanks for the help !
>>
>> Just waiting for some feedback before starting the "actions" ;)
>>
>> Regards
>> JB
>>
>>> Le 12 mars 2021 à 14:29, Jonathan Gallimore <
>> jonathan.gallimore@gmail.com> a écrit :
>>>
>>> I ran into this test failing yesterday:
>>>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>> - I'd be happy to try and contribute a fix. Would you like to assign the
>>> JIRA to me?
>>>
>>> Jon
>>>
>>> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> Now that we have Jenkinsfile in our repo, and we use Jenkins pipeline,
>> we
>>>> dramatically improved our build: the build is executed for each
>>>> PullRequests or commit on the main branch.
>>>>
>>>> However, we have lot of failing tests, causing quite systematically the
>>>> build failing on ci-builds.apache.org.
>>>>
>>>> We really need to have a clean, accurate and stable build: it will
>> improve
>>>> the issue detection and simplify the review, especially for
>> PullRequests.
>>>>
>>>> I ran several builds on my machine (with different docker containers)
>> and
>>>> I already identified some failing/flaky tests:
>>>>
>>>> -
>>>>
>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>>>> is not a big deal as I have a PR removing leveled completely
>>>> -
>>>>
>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>>>> Chris did an improvement, but I still have some flakiness here.
>>>> -
>>>>
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>>>
>>>> I propose the following action plan:
>>>>
>>>> 1. Create the Jira for each failing/flaky tests
>>>> 2. Exclude the tests (in surefire plugin configuration) to have a "green
>>>> light" on Jenkins.
>>>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins is
>>>> still "happy".
>>>>
>>>> Anyone willing to help on (3) is welcome !
>>>>
>>>> If there’s no objection, I will start with (1) and (2).
>>>>
>>>> Thanks,
>>>> Regards
>>>> JB
>>
>>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jonathan Gallimore <jo...@gmail.com>.
I'm +1 on the actions :).
Jon
On Fri, Mar 12, 2021 at 3:11 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
wrote:
> Sure, thanks for the help !
>
> Just waiting for some feedback before starting the "actions" ;)
>
> Regards
> JB
>
> > Le 12 mars 2021 à 14:29, Jonathan Gallimore <
> jonathan.gallimore@gmail.com> a écrit :
> >
> > I ran into this test failing yesterday:
> >
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> > - I'd be happy to try and contribute a fix. Would you like to assign the
> > JIRA to me?
> >
> > Jon
> >
> > On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
> > wrote:
> >
> >> Hi guys,
> >>
> >> Now that we have Jenkinsfile in our repo, and we use Jenkins pipeline,
> we
> >> dramatically improved our build: the build is executed for each
> >> PullRequests or commit on the main branch.
> >>
> >> However, we have lot of failing tests, causing quite systematically the
> >> build failing on ci-builds.apache.org.
> >>
> >> We really need to have a clean, accurate and stable build: it will
> improve
> >> the issue detection and simplify the review, especially for
> PullRequests.
> >>
> >> I ran several builds on my machine (with different docker containers)
> and
> >> I already identified some failing/flaky tests:
> >>
> >> -
> >>
> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
> >> is not a big deal as I have a PR removing leveled completely
> >> -
> >>
> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
> >> Chris did an improvement, but I still have some flakiness here.
> >> -
> >>
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> >>
> >> I propose the following action plan:
> >>
> >> 1. Create the Jira for each failing/flaky tests
> >> 2. Exclude the tests (in surefire plugin configuration) to have a "green
> >> light" on Jenkins.
> >> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins is
> >> still "happy".
> >>
> >> Anyone willing to help on (3) is welcome !
> >>
> >> If there’s no objection, I will start with (1) and (2).
> >>
> >> Thanks,
> >> Regards
> >> JB
>
>
Re: [PROPOSAL] Improve ActiveMQ 5 build stability
Posted by Jean-Baptiste Onofre <jb...@nanthrax.net>.
Sure, thanks for the help !
Just waiting for some feedback before starting the "actions" ;)
Regards
JB
> Le 12 mars 2021 à 14:29, Jonathan Gallimore <jo...@gmail.com> a écrit :
>
> I ran into this test failing yesterday:
> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
> - I'd be happy to try and contribute a fix. Would you like to assign the
> JIRA to me?
>
> Jon
>
> On Fri, Mar 12, 2021 at 12:58 PM Jean-Baptiste Onofre <jb...@nanthrax.net>
> wrote:
>
>> Hi guys,
>>
>> Now that we have Jenkinsfile in our repo, and we use Jenkins pipeline, we
>> dramatically improved our build: the build is executed for each
>> PullRequests or commit on the main branch.
>>
>> However, we have lot of failing tests, causing quite systematically the
>> build failing on ci-builds.apache.org.
>>
>> We really need to have a clean, accurate and stable build: it will improve
>> the issue detection and simplify the review, especially for PullRequests.
>>
>> I ran several builds on my machine (with different docker containers) and
>> I already identified some failing/flaky tests:
>>
>> -
>> activemq-leveldb-store/src/test/java/org/apache/activemq/leveldb/test/ElectingLevelDBStoreTest.java
>> is not a big deal as I have a PR removing leveled completely
>> -
>> activemq-stomp/src/test/java/org/apache/activemq/transport/stomp/Stomp11NIOSSLTest.java.
>> Chris did an improvement, but I still have some flakiness here.
>> -
>> activemq-unit-tests/src/test/java/org/apache/activemq/usecases/DuplexAdvisoryRaceTest.java
>>
>> I propose the following action plan:
>>
>> 1. Create the Jira for each failing/flaky tests
>> 2. Exclude the tests (in surefire plugin configuration) to have a "green
>> light" on Jenkins.
>> 3. For each Jira, we work on a PullRequest, to be sure that Jenkins is
>> still "happy".
>>
>> Anyone willing to help on (3) is welcome !
>>
>> If there’s no objection, I will start with (1) and (2).
>>
>> Thanks,
>> Regards
>> JB