You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Aristedes Maniatis <am...@apache.org> on 2016/09/10 01:52:20 UTC
Re: MS-SQL, mysql, etc
On 5/08/2016 10:51am, Aristedes Maniatis wrote:
> Apache Cayenne runs tests against a couple of in-memory databases under Jenkins:
>
> https://builds.apache.org/job/cayenne-master
>
>
> I'd like to set up tests against other common databases since database integration tests are absolutely critical to the Cayenne project. At the very least MS-SQL, mysql, mariadb and postgresql would be lovely. Even more would be better...
>
> Do we already have this sort of infrastructure available to us?
>
Is there any database infrastructure available for Jenkins builds?
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: MS-SQL, mysql, etc
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 13/09/2016 3:15pm, Stefan Seelmann wrote:
> On 09/13/2016 02:41 AM, Aristedes Maniatis wrote:
>> Do you know of any projects running on Jenkins I can look at that do something similar?
>
> For Apache Directory we currently have two builds [2][3] using Docker,
> but there we start the containers from a shell script.
>
>> I've actually never used docker and maven drives me up the wall (but not yet quite enough to put the effort into migrating Cayenne to gradle). So a bit of copy and paste would help.
>
> I sent you a pull request [4] that adds a new Maven profile which starts
> a Postgres Docker container before integration tests and stops it
> afterwards. For other DBs it should look similar.
>
> Side note: two tests in cayenne-server fail against Postgres:
> DataContextIteratedQueryIT.testPerformIteratedQuery_resolve:207 �
> CayenneRuntime
> DataContextIteratedQueryIT.testPerformIteratedQuery_CommitWithinIterator:227
> � CayenneRuntime
>
>> I am assuming I'll want to pull the Docker images from somewhere on local Apache infrastructure rather than downloading them from the internet for every build?
>
> No, the Docker image is downloaded on demand and stored/cached locally,
> so the next build just reuses the downloaded image.
>
> Kind Regards,
> Stefan
>
> [2] https://builds.apache.org/job/dir-fortress-core-docker-test/
> [3] https://builds.apache.org/job/dir-apacheds-docker-installers/
> [4] https://github.com/apache/cayenne/pull/111
>
Thanks so much for this Stefan. Much appreciated.
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: MS-SQL, mysql, etc
Posted by Jake Farrell <jf...@apache.org>.
Apache Mesos also uses Docker for its builds, they have a script [1] that
creates the dockerfile and builds out the container from there. The Apache
Thrift job also is an example of how you can setup to build/test using
Docker with jenkins, though we have switched to Travis for our primary CI
now.
If you have any questions happy to help
-Jake
[1]: https://github.com/apache/mesos/blob/master/support/docker_build.sh
On Tue, Sep 13, 2016 at 1:15 AM, Stefan Seelmann <ma...@stefan-seelmann.de>
wrote:
> On 09/13/2016 02:41 AM, Aristedes Maniatis wrote:
> > Do you know of any projects running on Jenkins I can look at that do
> something similar?
>
> For Apache Directory we currently have two builds [2][3] using Docker,
> but there we start the containers from a shell script.
>
> > I've actually never used docker and maven drives me up the wall (but not
> yet quite enough to put the effort into migrating Cayenne to gradle). So a
> bit of copy and paste would help.
>
> I sent you a pull request [4] that adds a new Maven profile which starts
> a Postgres Docker container before integration tests and stops it
> afterwards. For other DBs it should look similar.
>
> Side note: two tests in cayenne-server fail against Postgres:
> DataContextIteratedQueryIT.testPerformIteratedQuery_resolve:207 »
> CayenneRuntime
> DataContextIteratedQueryIT.testPerformIteratedQuery_
> CommitWithinIterator:227
> » CayenneRuntime
>
> > I am assuming I'll want to pull the Docker images from somewhere on
> local Apache infrastructure rather than downloading them from the internet
> for every build?
>
> No, the Docker image is downloaded on demand and stored/cached locally,
> so the next build just reuses the downloaded image.
>
> Kind Regards,
> Stefan
>
> [2] https://builds.apache.org/job/dir-fortress-core-docker-test/
> [3] https://builds.apache.org/job/dir-apacheds-docker-installers/
> [4] https://github.com/apache/cayenne/pull/111
>
>
Re: MS-SQL, mysql, etc
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 09/13/2016 02:41 AM, Aristedes Maniatis wrote:
> Do you know of any projects running on Jenkins I can look at that do something similar?
For Apache Directory we currently have two builds [2][3] using Docker,
but there we start the containers from a shell script.
> I've actually never used docker and maven drives me up the wall (but not yet quite enough to put the effort into migrating Cayenne to gradle). So a bit of copy and paste would help.
I sent you a pull request [4] that adds a new Maven profile which starts
a Postgres Docker container before integration tests and stops it
afterwards. For other DBs it should look similar.
Side note: two tests in cayenne-server fail against Postgres:
DataContextIteratedQueryIT.testPerformIteratedQuery_resolve:207 �
CayenneRuntime
DataContextIteratedQueryIT.testPerformIteratedQuery_CommitWithinIterator:227
� CayenneRuntime
> I am assuming I'll want to pull the Docker images from somewhere on local Apache infrastructure rather than downloading them from the internet for every build?
No, the Docker image is downloaded on demand and stored/cached locally,
so the next build just reuses the downloaded image.
Kind Regards,
Stefan
[2] https://builds.apache.org/job/dir-fortress-core-docker-test/
[3] https://builds.apache.org/job/dir-apacheds-docker-installers/
[4] https://github.com/apache/cayenne/pull/111
Re: MS-SQL, mysql, etc
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Sep 12, 2016, at 5:41 PM, Aristedes Maniatis <ar...@maniatis.org> wrote:
>
> Do you know of any projects running on Jenkins I can look at that do something similar? I've actually never used docker and maven drives me up the wall (but not yet quite enough to put the effort into migrating Cayenne to gradle). So a bit of copy and paste would help.
FWIW, this is a feature of Apache Yetus, which is a series of tools for other projects to use to do things like nightly builds (qbt) and patch testing (test-patch). The Docker features are being used by Hadoop and HBase and maybe a few others. We have a bundled Dockerfile that can be used as a starting point. Then as you customize it, you can have Yetus reference your project's Dockerfile from your source tree.
> I am assuming I'll want to pull the Docker images from somewhere on local Apache infrastructure rather than downloading them from the internet for every build?
Most of the Apache Yetus builds will build the image the first time on a given host from the Internet and then use the local cache from there. This allows one to test the actual Dockerfile as part of the build itself.
Re: MS-SQL, mysql, etc
Posted by Aristedes Maniatis <ar...@maniatis.org>.
On 10/09/2016 8:42pm, Stefan Seelmann wrote:
> On 09/10/2016 03:52 AM, Aristedes Maniatis wrote:
>> On 5/08/2016 10:51am, Aristedes Maniatis wrote:
>>> Apache Cayenne runs tests against a couple of in-memory databases under Jenkins:
>>>
>>> https://builds.apache.org/job/cayenne-master
>>>
>>>
>>> I'd like to set up tests against other common databases since database integration tests are absolutely critical to the Cayenne project. At the very least MS-SQL, mysql, mariadb and postgresql would be lovely. Even more would be better...
>>>
>>> Do we already have this sort of infrastructure available to us?
>>>
>>
>>
>> Is there any database infrastructure available for Jenkins builds?
>
> At least for the open source databases (mysql, mariadb, postgres) you
> can consider to use Docker, there already exists "official" Docker images.
>
> In your Jenkins job you can start e.g. a Postgres Docker container
> before the main build and stop/remove it after the build.
>
> Or, as Cayenne uses a Maven build: I made good experience with [1],
> that's a Maven plugin that starts a Docker container in
> pre-integration-test phase and stops it in post-integration-test phase.
>
> Kind Regards,
> Stefan
>
> [1] https://dmp.fabric8.io/
>
Thanks Stefan, I'll give it a go.
Do you know of any projects running on Jenkins I can look at that do something similar? I've actually never used docker and maven drives me up the wall (but not yet quite enough to put the effort into migrating Cayenne to gradle). So a bit of copy and paste would help.
I am assuming I'll want to pull the Docker images from somewhere on local Apache infrastructure rather than downloading them from the internet for every build?
Ari
--
-------------------------->
Aristedes Maniatis
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Re: MS-SQL, mysql, etc
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 09/10/2016 03:52 AM, Aristedes Maniatis wrote:
> On 5/08/2016 10:51am, Aristedes Maniatis wrote:
>> Apache Cayenne runs tests against a couple of in-memory databases under Jenkins:
>>
>> https://builds.apache.org/job/cayenne-master
>>
>>
>> I'd like to set up tests against other common databases since database integration tests are absolutely critical to the Cayenne project. At the very least MS-SQL, mysql, mariadb and postgresql would be lovely. Even more would be better...
>>
>> Do we already have this sort of infrastructure available to us?
>>
>
>
> Is there any database infrastructure available for Jenkins builds?
At least for the open source databases (mysql, mariadb, postgres) you
can consider to use Docker, there already exists "official" Docker images.
In your Jenkins job you can start e.g. a Postgres Docker container
before the main build and stop/remove it after the build.
Or, as Cayenne uses a Maven build: I made good experience with [1],
that's a Maven plugin that starts a Docker container in
pre-integration-test phase and stops it in post-integration-test phase.
Kind Regards,
Stefan
[1] https://dmp.fabric8.io/