You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Shawn McKinney <sm...@apache.org> on 2018/08/27 22:22:30 UTC
Working on dir-fortress-core-docker-openldap-test failures in jenkins
Howdy,
A couple of issues running fortress builds in jenkins on docker images. The openldap one is topic of this post.
Specifically, the problem with how the docker script runs:
excerpt from:
https://github.com/apache/directory-fortress-core/blob/master/src/docker/openldap-for-apache-fortress-tests/run-tests.sh
# configure build.properties
cp build.properties.example build.properties
# We workaround incorrect starting values by updating existing configs:
sed -i 's/^ldap\.server\.type=.*/ldap.server.type=openldap/' build.properties
sed -i 's/^ldap\.host=.*/ldap.host=localhost/' build.properties
sed -i 's/^ldap\.port=.*/ldap.port='${CONTAINER_PORT}'/' build.properties
sed -i 's/^suffix\.name=.*/suffix.name=example/' build.properties
sed -i 's/^suffix\.dc=.*/suffix.dc=com/' build.properties
sed -i 's/^root\.dn=.*/root.dn=cn=Manager,${suffix}/' build.properties
sed -i 's/^root\.pw=.*/root.pw={SSHA}pSOV2TpCxj2NMACijkcMko4fGrFopctU/' build.properties
sed -i 's/^cfg\.root\.pw=.*/cfg.root.pw=secret/‘ build.properties
This is all worked until a month or so ago, when I started refactoring the properties files, making sure that apacheds specific params were in build.properties, and openldap specific params in slapd.properties.
Now, the new problem, a few more properties need to be added, that aren’t in the build.properties.example, used as the starting point.
I tried updating the run-tests.sh (docker) script by appending the necessary props to the end of the build.properties file:
# a few more directives to seed additional properties:
echo "" >> build.properties
echo "log.admin.user=cn=Manager,cn=log" >> build.properties
echo "log.admin.pw=secret" >> build.properties
echo "min.log.conn=1" >> build.properties
echo "max.log.conn=3" >> build.properties
echo "audits.dn=cn=log" >> build.properties
From what I can tell, this hasn’t had any effect but I’ll admit to being ignorant how the jenkins tests bootstrap the docker images. Does it call the run-tests.sh script, or is there another script somewhere, that must be changed?
I can workaround this issue but adding the openldap specific props *back* into the build.properties.example, but that isn’t consistent usage.
The best solution would be to change the docker script to use the slapd.properties.example as the base, which then would reduce the number of updates that have to be done using sed or echo commands.
Anyway, apologize for the lengthy post. I’ll keep poking around and let y’all know what I find out.
L8tr,
—Arkanshawn
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Shawn McKinney <sm...@apache.org>.
>
> On Sep 9, 2018, at 1:04 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> Cool, thanks.
>
> I replaced both, the OpenLDAP and ApacheDS one. The OpenLDAP runs stable
> now.
Very nice.
> On Sep 9, 2018, at 1:04 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> However the test that uses ApacheDS always fails. Either it times out
> after 2 hours (!) or it fails with a password expired message. Locally
> the test runs successful in approx. 30 minutes. I assume the tests do a
> lot of I/O and fsync is enabled in ApacheDS and the Jenkins servers have
> only spindle disks. I disabled the test for now and continue to
> investigate after the Studio release.
Yeah, I made the same observation, thanks for brining it up.
I can run the apacheds docker tests, on a local VM, in about 15 minutes. There is certainly a lot of i/o, thousands of entries are CRUDed, but even 30 minutes is way too long, two hours -- something’s wrong. Strange that these tests used to pass, some change that I’ve made must have broke them.
It’s on my todo list…
Thanks,
—Shawn
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 9/8/18 12:51 PM, Shawn McKinney wrote:
>
>> On Sep 8, 2018, at 2:00 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>>
>>
>> I wanted to do that now, but now the src/docker and other config folders
>> are gone, was that on purpose?
>
> Negative, was a mistake, on my part. I’ve had some strange effects in my dev VM. Anyway these files/folders have been re-added to the project repo. I’ll test on a clean VM later today to verify I didn’t miss anything. In the meantime, please proceed when ready.
Cool, thanks.
I replaced both, the OpenLDAP and ApacheDS one. The OpenLDAP runs stable
now.
However the test that uses ApacheDS always fails. Either it times out
after 2 hours (!) or it fails with a password expired message. Locally
the test runs successful in approx. 30 minutes. I assume the tests do a
lot of I/O and fsync is enabled in ApacheDS and the Jenkins servers have
only spindle disks. I disabled the test for now and continue to
investigate after the Studio release.
Kind Regards,
Stefan
[1]
https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-apacheds-test/6/console
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Shawn McKinney <sm...@apache.org>.
> On Sep 8, 2018, at 2:00 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
>
> I wanted to do that now, but now the src/docker and other config folders
> are gone, was that on purpose?
Negative, was a mistake, on my part. I’ve had some strange effects in my dev VM. Anyway these files/folders have been re-added to the project repo. I’ll test on a clean VM later today to verify I didn’t miss anything. In the meantime, please proceed when ready.
Thanks
—Shawn
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
>> I created a 2nd one for now, which works when setting maven and java
>> home env variables (there is no global maven configured which is fine)
>> https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test-2
>>
>> If you want we can delete the old one and rename this. Using the old one
>> won't work because that's a "Maven" job type while the other is just a
>> "Freestyle" job type.
>
> A new test is fine, assuming the old one is disabled. Or, we can rename it. Either way works for me.
I wanted to do that now, but now the src/docker and other config folders
are gone, was that on purpose?
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Shawn McKinney <sm...@apache.org>.
> On Aug 28, 2018, at 1:27 PM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
>> https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test/configure
>>
>> It doesn't use the run-tests.sh script at all. All the instructions are
>> duplicated there. I guess I did that, not a nice move, but I don't
>> remember why...
>>
>> Probably the easiest change is to use that script in Jenkins, I'll try..
>
> I created a 2nd one for now, which works when setting maven and java
> home env variables (there is no global maven configured which is fine)
> https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test-2
>
> If you want we can delete the old one and rename this. Using the old one
> won't work because that's a "Maven" job type while the other is just a
> "Freestyle" job type.
A new test is fine, assuming the old one is disabled. Or, we can rename it. Either way works for me.
>
> On Aug 28, 2018, at 1:27 PM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> Using the shell script is not optimal because it may leave back the
> running OpenLDAP container in case something fails in the middle. I
> think the cleanest solution would be to specify the build in a
> Jenkinsfile. I started to create one for the LDAP API and the server,
> but they need to be reworked, I'm still learning.
Agreed. Anything I can do to help?
—Shawn
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
> https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test/configure
>
> It doesn't use the run-tests.sh script at all. All the instructions are
> duplicated there. I guess I did that, not a nice move, but I don't
> remember why...
>
> Probably the easiest change is to use that script in Jenkins, I'll try..
I created a 2nd one for now, which works when setting maven and java
home env variables (there is no global maven configured which is fine)
https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test-2
If you want we can delete the old one and rename this. Using the old one
won't work because that's a "Maven" job type while the other is just a
"Freestyle" job type.
Using the shell script is not optimal because it may leave back the
running OpenLDAP container in case something fails in the middle. I
think the cleanest solution would be to specify the build in a
Jenkinsfile. I started to create one for the LDAP API and the server,
but they need to be reworked, I'm still learning.
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 08/28/2018 02:46 PM, Shawn McKinney wrote:
>> Please have a look into the Jenkin job configuration:
>> https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test/configure
>
> Ah yes that’s is what I figured. Btw, when I go to that page, I get:
>
> 'Access Denied'
>
> Perhaps I can get access?
Sure, done.
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Shawn McKinney <sm...@apache.org>.
> On Aug 28, 2018, at 12:17 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> Please have a look into the Jenkin job configuration:
> https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test/configure
Ah yes that’s is what I figured. Btw, when I go to that page, I get:
'Access Denied'
Perhaps I can get access?
>
> On Aug 28, 2018, at 12:17 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
> It doesn't use the run-tests.sh script at all. All the instructions are
> duplicated there. I guess I did that, not a nice move, but I don't
> remember why...
>
> Probably the easiest change is to use that script in Jenkins, I'll try..
>
> Btw, when you run the script locally does it succeed?
Yes
>
> On Aug 28, 2018, at 12:17 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
>> The best solution would be to change the docker script to use the slapd.properties.example as the base, which then would reduce the number of updates that have to be done using sed or echo commands.
>
> Sure, just replace the properties file name in the script
Done, checked into latest.
>
> On Aug 28, 2018, at 12:17 AM, Stefan Seelmann <ma...@stefan-seelmann.de> wrote:
>
>> Anyway, apologize for the lengthy post. I’ll keep poking around and let y’all know what I find out.
>
> No, all fine, I have to apologize for not looking into the issue earlier...
No worries Stefan. This has helped me to learn more about Docker and becoming less mysterious.
Thanks for responding.
—Arkanshawn
Re: Working on dir-fortress-core-docker-openldap-test failures in
jenkins
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
Hi Shawn,
On 08/28/2018 12:22 AM, Shawn McKinney wrote:
> Howdy,
>
> A couple of issues running fortress builds in jenkins on docker images. The openldap one is topic of this post.
>
> Specifically, the problem with how the docker script runs:
>
> excerpt from:
> https://github.com/apache/directory-fortress-core/blob/master/src/docker/openldap-for-apache-fortress-tests/run-tests.sh
>
>
> # configure build.properties
> cp build.properties.example build.properties
>
> # We workaround incorrect starting values by updating existing configs:
> sed -i 's/^ldap\.server\.type=.*/ldap.server.type=openldap/' build.properties
> sed -i 's/^ldap\.host=.*/ldap.host=localhost/' build.properties
> sed -i 's/^ldap\.port=.*/ldap.port='${CONTAINER_PORT}'/' build.properties
> sed -i 's/^suffix\.name=.*/suffix.name=example/' build.properties
> sed -i 's/^suffix\.dc=.*/suffix.dc=com/' build.properties
> sed -i 's/^root\.dn=.*/root.dn=cn=Manager,${suffix}/' build.properties
> sed -i 's/^root\.pw=.*/root.pw={SSHA}pSOV2TpCxj2NMACijkcMko4fGrFopctU/' build.properties
> sed -i 's/^cfg\.root\.pw=.*/cfg.root.pw=secret/‘ build.properties
>
> This is all worked until a month or so ago, when I started refactoring the properties files, making sure that apacheds specific params were in build.properties, and openldap specific params in slapd.properties.
>
> Now, the new problem, a few more properties need to be added, that aren’t in the build.properties.example, used as the starting point.
>
> I tried updating the run-tests.sh (docker) script by appending the necessary props to the end of the build.properties file:
>
> # a few more directives to seed additional properties:
> echo "" >> build.properties
> echo "log.admin.user=cn=Manager,cn=log" >> build.properties
> echo "log.admin.pw=secret" >> build.properties
> echo "min.log.conn=1" >> build.properties
> echo "max.log.conn=3" >> build.properties
> echo "audits.dn=cn=log" >> build.properties
>
> From what I can tell, this hasn’t had any effect but I’ll admit to being ignorant how the jenkins tests bootstrap the docker images. Does it call the run-tests.sh script, or is there another script somewhere, that must be changed?
Please have a look into the Jenkin job configuration:
https://builds.apache.org/view/D/view/Directory/job/dir-fortress-core-docker-openldap-test/configure
It doesn't use the run-tests.sh script at all. All the instructions are
duplicated there. I guess I did that, not a nice move, but I don't
remember why...
Probably the easiest change is to use that script in Jenkins, I'll try..
Btw, when you run the script locally does it succeed?
> The best solution would be to change the docker script to use the slapd.properties.example as the base, which then would reduce the number of updates that have to be done using sed or echo commands.
Sure, just replace the properties file name in the script
> Anyway, apologize for the lengthy post. I’ll keep poking around and let y’all know what I find out.
No, all fine, I have to apologize for not looking into the issue earlier...