Further itest speed improvements

[Christian Schneider <ch...@die-schneider.net>]

I would like to discuss some ideas to further speed up our itests.

Ideas:
- Split our jenkins builds into itest and deploy builds: On a local 
machine the itest build finishs now in about 10 minutes. On jenkins the 
build still takes over 40 minutes. Most of the time is spent on doing 
the deploy part. So I propose to have two builds for each branch. One 
itest build for fast developer feedback that runs after each commit and 
one deploy build to keep the snapshots current that only runs once per day.
- Use PerClass for the feature tests and uninstall each feature before 
the next test method runs. This brings the full build down to 5 minutes 
on my machine. It works quite well. I will commit this if the tests are 
stable on my machine.
- Use PerSuite for all non destructive tests (basically all tests that 
should not change the karaf instance they run in). Pax exam now allows 
to start the container only once for a whole suite of tests. We probably 
need to move these tests in a separate maven project. This should 
further speed up the tests quite a lot as most of our tests are non 
destructive. I will have to do some more tests about this.

So what do you think?

Christian


-- 
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com

Re: Further itest speed improvements

[David Bosschaert <da...@gmail.com>]

As an additional note, I found that some changes to the itests that I
introduced with the RBAC for shell commands don't run as quick as I
expected them to run. I'm planning to look into this, but it probably
won't be before next week...

Cheers,

David

On 27 October 2013 12:48, Achim Nierbeck <bc...@googlemail.com> wrote:
> Hi guys,
>
> I'm with ya' that a nightly with deploy should be sufficient.
> Regarding the PerSuite / other Speed enhancements I'm not against but
> we'd need to be very cautions that we don't have side-effects of this.
> That's why I usually prefer to have the perTest fork and/or annotation on
> the UnitTests.
> It just makes sure you work on a clean container (just like you would work
> on a clean database ;) )
> I think at that point we just need to "bite that bullet" and keep to those
> longer running tests.
>
>
> regards, Achim
>
>
>
> 2013/10/27 Jean-Baptiste Onofré <jb...@nanthrax.net>
>
>> Hi Christian,
>>
>> It's what I said in my previous e-mail: if possible, I would like to avoid
>> a job dedicated to itests. If there are no other ways, we will do like this.
>>
>> For the PerSuite, you have an option in Pax Exam to reuse an existing
>> container. Let me try with that.
>>
>> Regards
>> JB
>>
>>
>> On 10/27/2013 10:35 AM, Christian Schneider wrote:
>>
>>> Hi JB,
>>>
>>> I created an itest build here:
>>> https://builds.apache.org/**view/H-L/view/Karaf/job/Karaf-**itests/<https://builds.apache.org/view/H-L/view/Karaf/job/Karaf-itests/>
>>>
>>> You can reuse it or delete it when you create a new build.
>>>
>>> About the PerSuite tests. I have experimented a bit. In eclipse I was
>>> able to run most of the karaf tests successfully in PerSuite mode and
>>> they are really extremely fast. Unfortunately I was not able to make
>>> them work with maven. In maven the karaf container was always created
>>> for each test class. I think it is related to the fork mode of surefire.
>>> By default it seems to start a fresh process for each test class. When I
>>> changed forkCount to 0 none of the tests work. So the results of my
>>> experiments is that it has a lot of potential but currently does not
>>> work correctly.
>>>
>>> Christian
>>>
>>> Am 27.10.2013 07:49, schrieb Jean-Baptiste Onofré:
>>>
>>>> Hi Christian,
>>>>
>>>> On 10/26/2013 09:28 PM, Christian Schneider wrote:
>>>>
>>>>> On 26.10.2013 21:14, Jean-Baptiste Onofré wrote:
>>>>>
>>>>>> Hi Christian,
>>>>>>
>>>>>> - If you mean create two job in Jenkins, I disagree. I would prefer to
>>>>>> work with profiles. More over, it makes sense to tight the itest on
>>>>>> the artifacts that we just built before. Maybe we can have to target
>>>>>> on Jenkins: one with the itest profile, one without the itest profile.
>>>>>> The trigger for the full build (including itest) is every night, the
>>>>>> build without itest is at scm polling.
>>>>>>
>>>>> I did not want to have a build without itests. The itests are fast
>>>>> enough to run in both builds. I wanted one build with itests after each
>>>>> commit and one build with itests and deploy nightly. The problem
>>>>> currently seems to be the deploy it takes much longer than the tests.
>>>>>
>>>> Catcha, so build without deploy after scm poll, and deploy only for
>>>> nightly builds ? I can do that.
>>>>
>>>>  - For the PerClass test on feature, I'm agree.
>>>>>> - Not sure I follow you for the PerSuite. What do you mean ?
>>>>>>
>>>>> PerSuite will start karaf only once and run several test classes on it.
>>>>>
>>>> Catcha, you mean reuse the same Karaf container for all itests. We
>>>> just have to be careful that one itest doesn't impact another. I have
>>>> issues like this when I implemented itests.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>>
>>>>> Christian
>>>>>
>>>>>
>>>>
>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>
>
> --
>
> Apache Karaf <http://karaf.apache.org/> Committer & PMC
> OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer &
> Project Lead
> OPS4J Pax for Vaadin <http://team.ops4j.org/wiki/display/PAXVAADIN/Home>
> Commiter & Project Lead
> blog <http://notizblog.nierbeck.de/>

Re: Further itest speed improvements

[Achim Nierbeck <bc...@googlemail.com>]

Hi guys,

I'm with ya' that a nightly with deploy should be sufficient.
Regarding the PerSuite / other Speed enhancements I'm not against but
we'd need to be very cautions that we don't have side-effects of this.
That's why I usually prefer to have the perTest fork and/or annotation on
the UnitTests.
It just makes sure you work on a clean container (just like you would work
on a clean database ;) )
I think at that point we just need to "bite that bullet" and keep to those
longer running tests.


regards, Achim



2013/10/27 Jean-Baptiste Onofré <jb...@nanthrax.net>

> Hi Christian,
>
> It's what I said in my previous e-mail: if possible, I would like to avoid
> a job dedicated to itests. If there are no other ways, we will do like this.
>
> For the PerSuite, you have an option in Pax Exam to reuse an existing
> container. Let me try with that.
>
> Regards
> JB
>
>
> On 10/27/2013 10:35 AM, Christian Schneider wrote:
>
>> Hi JB,
>>
>> I created an itest build here:
>> https://builds.apache.org/**view/H-L/view/Karaf/job/Karaf-**itests/<https://builds.apache.org/view/H-L/view/Karaf/job/Karaf-itests/>
>>
>> You can reuse it or delete it when you create a new build.
>>
>> About the PerSuite tests. I have experimented a bit. In eclipse I was
>> able to run most of the karaf tests successfully in PerSuite mode and
>> they are really extremely fast. Unfortunately I was not able to make
>> them work with maven. In maven the karaf container was always created
>> for each test class. I think it is related to the fork mode of surefire.
>> By default it seems to start a fresh process for each test class. When I
>> changed forkCount to 0 none of the tests work. So the results of my
>> experiments is that it has a lot of potential but currently does not
>> work correctly.
>>
>> Christian
>>
>> Am 27.10.2013 07:49, schrieb Jean-Baptiste Onofré:
>>
>>> Hi Christian,
>>>
>>> On 10/26/2013 09:28 PM, Christian Schneider wrote:
>>>
>>>> On 26.10.2013 21:14, Jean-Baptiste Onofré wrote:
>>>>
>>>>> Hi Christian,
>>>>>
>>>>> - If you mean create two job in Jenkins, I disagree. I would prefer to
>>>>> work with profiles. More over, it makes sense to tight the itest on
>>>>> the artifacts that we just built before. Maybe we can have to target
>>>>> on Jenkins: one with the itest profile, one without the itest profile.
>>>>> The trigger for the full build (including itest) is every night, the
>>>>> build without itest is at scm polling.
>>>>>
>>>> I did not want to have a build without itests. The itests are fast
>>>> enough to run in both builds. I wanted one build with itests after each
>>>> commit and one build with itests and deploy nightly. The problem
>>>> currently seems to be the deploy it takes much longer than the tests.
>>>>
>>> Catcha, so build without deploy after scm poll, and deploy only for
>>> nightly builds ? I can do that.
>>>
>>>  - For the PerClass test on feature, I'm agree.
>>>>> - Not sure I follow you for the PerSuite. What do you mean ?
>>>>>
>>>> PerSuite will start karaf only once and run several test classes on it.
>>>>
>>> Catcha, you mean reuse the same Karaf container for all itests. We
>>> just have to be careful that one itest doesn't impact another. I have
>>> issues like this when I implemented itests.
>>>
>>> Regards
>>> JB
>>>
>>>
>>>> Christian
>>>>
>>>>
>>>
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>



-- 

Apache Karaf <http://karaf.apache.org/> Committer & PMC
OPS4J Pax Web <http://wiki.ops4j.org/display/paxweb/Pax+Web/> Committer &
Project Lead
OPS4J Pax for Vaadin <http://team.ops4j.org/wiki/display/PAXVAADIN/Home>
Commiter & Project Lead
blog <http://notizblog.nierbeck.de/>

Re: Further itest speed improvements

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Christian,

It's what I said in my previous e-mail: if possible, I would like to 
avoid a job dedicated to itests. If there are no other ways, we will do 
like this.

For the PerSuite, you have an option in Pax Exam to reuse an existing 
container. Let me try with that.

Regards
JB

On 10/27/2013 10:35 AM, Christian Schneider wrote:
> Hi JB,
>
> I created an itest build here:
> https://builds.apache.org/view/H-L/view/Karaf/job/Karaf-itests/
>
> You can reuse it or delete it when you create a new build.
>
> About the PerSuite tests. I have experimented a bit. In eclipse I was
> able to run most of the karaf tests successfully in PerSuite mode and
> they are really extremely fast. Unfortunately I was not able to make
> them work with maven. In maven the karaf container was always created
> for each test class. I think it is related to the fork mode of surefire.
> By default it seems to start a fresh process for each test class. When I
> changed forkCount to 0 none of the tests work. So the results of my
> experiments is that it has a lot of potential but currently does not
> work correctly.
>
> Christian
>
> Am 27.10.2013 07:49, schrieb Jean-Baptiste Onofré:
>> Hi Christian,
>>
>> On 10/26/2013 09:28 PM, Christian Schneider wrote:
>>> On 26.10.2013 21:14, Jean-Baptiste Onofré wrote:
>>>> Hi Christian,
>>>>
>>>> - If you mean create two job in Jenkins, I disagree. I would prefer to
>>>> work with profiles. More over, it makes sense to tight the itest on
>>>> the artifacts that we just built before. Maybe we can have to target
>>>> on Jenkins: one with the itest profile, one without the itest profile.
>>>> The trigger for the full build (including itest) is every night, the
>>>> build without itest is at scm polling.
>>> I did not want to have a build without itests. The itests are fast
>>> enough to run in both builds. I wanted one build with itests after each
>>> commit and one build with itests and deploy nightly. The problem
>>> currently seems to be the deploy it takes much longer than the tests.
>> Catcha, so build without deploy after scm poll, and deploy only for
>> nightly builds ? I can do that.
>>
>>>> - For the PerClass test on feature, I'm agree.
>>>> - Not sure I follow you for the PerSuite. What do you mean ?
>>> PerSuite will start karaf only once and run several test classes on it.
>> Catcha, you mean reuse the same Karaf container for all itests. We
>> just have to be careful that one itest doesn't impact another. I have
>> issues like this when I implemented itests.
>>
>> Regards
>> JB
>>
>>>
>>> Christian
>>>
>>
>
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Further itest speed improvements

[Christian Schneider <ch...@die-schneider.net>]

Hi JB,

I created an itest build here:
https://builds.apache.org/view/H-L/view/Karaf/job/Karaf-itests/

You can reuse it or delete it when you create a new build.

About the PerSuite tests. I have experimented a bit. In eclipse I was 
able to run most of the karaf tests successfully in PerSuite mode and 
they are really extremely fast. Unfortunately I was not able to make 
them work with maven. In maven the karaf container was always created 
for each test class. I think it is related to the fork mode of surefire. 
By default it seems to start a fresh process for each test class. When I 
changed forkCount to 0 none of the tests work. So the results of my 
experiments is that it has a lot of potential but currently does not 
work correctly.

Christian

Am 27.10.2013 07:49, schrieb Jean-Baptiste Onofré:
> Hi Christian,
>
> On 10/26/2013 09:28 PM, Christian Schneider wrote:
>> On 26.10.2013 21:14, Jean-Baptiste Onofré wrote:
>>> Hi Christian,
>>>
>>> - If you mean create two job in Jenkins, I disagree. I would prefer to
>>> work with profiles. More over, it makes sense to tight the itest on
>>> the artifacts that we just built before. Maybe we can have to target
>>> on Jenkins: one with the itest profile, one without the itest profile.
>>> The trigger for the full build (including itest) is every night, the
>>> build without itest is at scm polling.
>> I did not want to have a build without itests. The itests are fast
>> enough to run in both builds. I wanted one build with itests after each
>> commit and one build with itests and deploy nightly. The problem
>> currently seems to be the deploy it takes much longer than the tests.
> Catcha, so build without deploy after scm poll, and deploy only for 
> nightly builds ? I can do that.
>
>>> - For the PerClass test on feature, I'm agree.
>>> - Not sure I follow you for the PerSuite. What do you mean ?
>> PerSuite will start karaf only once and run several test classes on it.
> Catcha, you mean reuse the same Karaf container for all itests. We 
> just have to be careful that one itest doesn't impact another. I have 
> issues like this when I implemented itests.
>
> Regards
> JB
>
>>
>> Christian
>>
>


-- 
  
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
Talend Application Integration Division http://www.talend.com

Re: Further itest speed improvements

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Christian,

On 10/26/2013 09:28 PM, Christian Schneider wrote:
> On 26.10.2013 21:14, Jean-Baptiste Onofré wrote:
>> Hi Christian,
>>
>> - If you mean create two job in Jenkins, I disagree. I would prefer to
>> work with profiles. More over, it makes sense to tight the itest on
>> the artifacts that we just built before. Maybe we can have to target
>> on Jenkins: one with the itest profile, one without the itest profile.
>> The trigger for the full build (including itest) is every night, the
>> build without itest is at scm polling.
> I did not want to have a build without itests. The itests are fast
> enough to run in both builds. I wanted one build with itests after each
> commit and one build with itests and deploy nightly. The problem
> currently seems to be the deploy it takes much longer than the tests.
Catcha, so build without deploy after scm poll, and deploy only for 
nightly builds ? I can do that.

>> - For the PerClass test on feature, I'm agree.
>> - Not sure I follow you for the PerSuite. What do you mean ?
> PerSuite will start karaf only once and run several test classes on it.
Catcha, you mean reuse the same Karaf container for all itests. We just 
have to be careful that one itest doesn't impact another. I have issues 
like this when I implemented itests.

Regards
JB

>
> Christian
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Further itest speed improvements

[Christian Schneider <ch...@die-schneider.net>]

On 26.10.2013 21:14, Jean-Baptiste Onofré wrote:
> Hi Christian,
>
> - If you mean create two job in Jenkins, I disagree. I would prefer to 
> work with profiles. More over, it makes sense to tight the itest on 
> the artifacts that we just built before. Maybe we can have to target 
> on Jenkins: one with the itest profile, one without the itest profile. 
> The trigger for the full build (including itest) is every night, the 
> build without itest is at scm polling.
I did not want to have a build without itests. The itests are fast 
enough to run in both builds. I wanted one build with itests after each 
commit and one build with itests and deploy nightly. The problem 
currently seems to be the deploy it takes much longer than the tests.
> - For the PerClass test on feature, I'm agree.
> - Not sure I follow you for the PerSuite. What do you mean ?
PerSuite will start karaf only once and run several test classes on it.

Christian

-- 
Christian Schneider
http://www.liquid-reality.de

Open Source Architect
http://www.talend.com

Re: Further itest speed improvements

[Jean-Baptiste Onofré <jb...@nanthrax.net>]

Hi Christian,

- If you mean create two job in Jenkins, I disagree. I would prefer to 
work with profiles. More over, it makes sense to tight the itest on the 
artifacts that we just built before. Maybe we can have to target on 
Jenkins: one with the itest profile, one without the itest profile. The 
trigger for the full build (including itest) is every night, the build 
without itest is at scm polling.
- For the PerClass test on feature, I'm agree.
- Not sure I follow you for the PerSuite. What do you mean ?

Regards
JB

On 10/26/2013 08:51 PM, Christian Schneider wrote:
> I would like to discuss some ideas to further speed up our itests.
>
> Ideas:
> - Split our jenkins builds into itest and deploy builds: On a local
> machine the itest build finishs now in about 10 minutes. On jenkins the
> build still takes over 40 minutes. Most of the time is spent on doing
> the deploy part. So I propose to have two builds for each branch. One
> itest build for fast developer feedback that runs after each commit and
> one deploy build to keep the snapshots current that only runs once per day.
> - Use PerClass for the feature tests and uninstall each feature before
> the next test method runs. This brings the full build down to 5 minutes
> on my machine. It works quite well. I will commit this if the tests are
> stable on my machine.
> - Use PerSuite for all non destructive tests (basically all tests that
> should not change the karaf instance they run in). Pax exam now allows
> to start the container only once for a whole suite of tests. We probably
> need to move these tests in a separate maven project. This should
> further speed up the tests quite a lot as most of our tests are non
> destructive. I will have to do some more tests about this.
>
> So what do you think?
>
> Christian
>
>

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com