You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Shawn McKinney <sm...@apache.org> on 2019/04/17 19:31:22 UTC
Working on Fortress Docker Images / Builds / Jenkins tests
Hi,
The docker images used by fortress to test openldap and apacheDS have been recently updated to support a minor change in the data schema, to support additional configuration params.
https://hub.docker.com/r/apachedirectory/openldap-for-apache-fortress-tests/
https://hub.docker.com/r/apachedirectory/apacheds-for-apache-fortress-tests/
I also cleaned up the docker build steps, removing extraneous data loads, paying special attention to an apacheds test that were failing / timing out.
As previously observed here, the apacheDS test takes an extraordinary amount of time to complete inside of Jenkins, and were disabled some time ago:
https://builds.apache.org/job/dir-fortress-core-docker-apacheds-test/
That is to say, that running the test on a local VM finishes in < 10 minutes, while in a Jenkins pipeline much longer, > 1 hour.
I’m attempting to work around this issue by removing a second run of the junit, one that performs a teardown, before reloading.
I’m currently verifying that these tweaks will enable the tests to run successfully.
Please let me know if you have any questions or concerns.
Thanks,
—Shawn
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 23, 2019, at 12:28 PM, Shawn McKinney <sm...@apache.org> wrote:
>
> Now running the job and it seems to be doing the right thing. Stay tuned, will report back here where this ends up.
Hello,
A few more days, many successful test runs and we can say this issue has been resolved.
A combination of removing the teardown phase at the end of script and blacklisting slow machines has done the trick.
—Shawn
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 23, 2019, at 12:21 PM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> I got an error because between H33 and H34 there is only on |. after
> fixing it saved. maybe try again?
>
> To get Jenkins access I only know [1], you can check yourself if you are
> in that group ;)
After correcting was able to save it, go figure. :-)
Now running the job and it seems to be doing the right thing. Stay tuned, will report back here where this ends up.
And many thanks for the help Stefan.
—Shawn
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 4/23/19 4:42 PM, Shawn McKinney wrote:
>
>> On Apr 23, 2019, at 9:02 AM, Shawn McKinney <sm...@apache.org> wrote:
>>
>> Looking at the job config, can we try something like this?
>>
>> "Restrict where this project can be run”
>>
>> ubuntu && !(H2% || H3%)
>>
>> That is are wildcards supported? Couldn’t figure it out by reading the help on the job website.
>>
>> Otherwise we’re looking at something like:
>>
>> ubuntu && !(H22 || H23 || H24 || H25 || H26 || H27 || H28 || H29 || H30 || H31 || H32 || H33 | H34 || H35 || H36 || H37 || H38)
>
> Whoops… a 403 error updating the 'Label Expression' field on this job.
>
> How does one get a bit more karma? Not sure I can handle the power. ;-)
I got an error because between H33 and H34 there is only on |. after
fixing it saved. maybe try again?
To get Jenkins access I only know [1], you can check yourself if you are
in that group ;)
Kind Regards,
Stefan
[1]
https://cwiki.apache.org/confluence/display/INFRA/Jenkins#Jenkins-HowdoIgetanaccount
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 23, 2019, at 9:02 AM, Shawn McKinney <sm...@apache.org> wrote:
>
> Looking at the job config, can we try something like this?
>
> "Restrict where this project can be run”
>
> ubuntu && !(H2% || H3%)
>
> That is are wildcards supported? Couldn’t figure it out by reading the help on the job website.
>
> Otherwise we’re looking at something like:
>
> ubuntu && !(H22 || H23 || H24 || H25 || H26 || H27 || H28 || H29 || H30 || H31 || H32 || H33 | H34 || H35 || H36 || H37 || H38)
Whoops… a 403 error updating the 'Label Expression' field on this job.
How does one get a bit more karma? Not sure I can handle the power. ;-)
—Shawn
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 21, 2019, at 2:22 PM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> Some more observations when I run the ApacheDS pipeline:
>
> Some builds finish in only 15-20 minutes, those run on host ubuntu-eu2
> or ubuntu-eu3. However builds on one of the H2x or H3x nodes take 40-60
> minutes.
Hi Stefan,
Noticed today another failed build or two, again on time sensitive ops.
Looking at the logs, H38 was the machine it was running on.
So, would like to try blacklisting the slower H2X and H3X nodes.
Looking at the job config, can we try something like this?
"Restrict where this project can be run”
ubuntu && !(H2% || H3%)
That is are wildcards supported? Couldn’t figure it out by reading the help on the job website.
Otherwise we’re looking at something like:
ubuntu && !(H22 || H23 || H24 || H25 || H26 || H27 || H28 || H29 || H30 || H31 || H32 || H33 | H34 || H35 || H36 || H37 || H38)
Which isn’t all that bad.
—Shawn
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 4/19/19 2:42 PM, Shawn McKinney wrote:
>
>> On Apr 18, 2019, at 3:49 PM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>>
>> Yes, the Jenkins machines are rather slow, I agree.
>>
>> There is a bit info (memory and disk) in the Jenkins wiki [1], our jobs
>> use the "ubuntu" label. For further info we need ask builds@a.o list.
>>
>> What's also possible, if you determine a slow node, to blacklist it in
>> the job's config "Restrict where this project can be run", like "ubuntu
>> && !H30". Or even only whitelist fast ones, but then the queue/wait time
>> may be higher.
>
> Hi Stefan, thanks for weighing in. This makes sense. Let me do some more experimentation.
>
> I’m also thinking about changing the junit tests themselves to not run the pwpolicy checks when running ADS in a Jenkins pipeline.
>
> Should be easy enough by looking at the env variables.
>
> Will let you know where we end up.
Some more observations when I run the ApacheDS pipeline:
Some builds finish in only 15-20 minutes, those run on host ubuntu-eu2
or ubuntu-eu3. However builds on one of the H2x or H3x nodes take 40-60
minutes.
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 18, 2019, at 3:49 PM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> Yes, the Jenkins machines are rather slow, I agree.
>
> There is a bit info (memory and disk) in the Jenkins wiki [1], our jobs
> use the "ubuntu" label. For further info we need ask builds@a.o list.
>
> What's also possible, if you determine a slow node, to blacklist it in
> the job's config "Restrict where this project can be run", like "ubuntu
> && !H30". Or even only whitelist fast ones, but then the queue/wait time
> may be higher.
Hi Stefan, thanks for weighing in. This makes sense. Let me do some more experimentation.
I’m also thinking about changing the junit tests themselves to not run the pwpolicy checks when running ADS in a Jenkins pipeline.
Should be easy enough by looking at the env variables.
Will let you know where we end up.
—Shawn
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 4/18/19 4:12 PM, Shawn McKinney wrote:
>
>> On Apr 17, 2019, at 2:31 PM, Shawn McKinney <sm...@apache.org> wrote:
>>
>> That is to say, that running the test on a local VM finishes in < 10 minutes, while in a Jenkins pipeline much longer, > 1 hour.
>
> It feels like the tests are running inside a VM that’s underpowered. The adds, updates and deletes are very slow. This is not a blocker, just causes the tests to take longer than usual.
>
> What is a blocker, the pw policy validations are time sensitive, and fail in random places.
>
> Can someone provide the specs on the VM’s that are allocated to run this test process?
Yes, the Jenkins machines are rather slow, I agree.
There is a bit info (memory and disk) in the Jenkins wiki [1], our jobs
use the "ubuntu" label. For further info we need ask builds@a.o list.
What's also possible, if you determine a slow node, to blacklist it in
the job's config "Restrict where this project can be run", like "ubuntu
&& !H30". Or even only whitelist fast ones, but then the queue/wait time
may be higher.
Kind Regards,
Stefan
[1] https://cwiki.apache.org/confluence/display/INFRA/Jenkins+node+labels
Re: Working on Fortress Docker Images / Builds / Jenkins tests
Posted by Shawn McKinney <sm...@apache.org>.
> On Apr 17, 2019, at 2:31 PM, Shawn McKinney <sm...@apache.org> wrote:
>
> That is to say, that running the test on a local VM finishes in < 10 minutes, while in a Jenkins pipeline much longer, > 1 hour.
It feels like the tests are running inside a VM that’s underpowered. The adds, updates and deletes are very slow. This is not a blocker, just causes the tests to take longer than usual.
What is a blocker, the pw policy validations are time sensitive, and fail in random places.
Can someone provide the specs on the VM’s that are allocated to run this test process?
Thanks,
—Shawn