You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Luca Toscano <to...@gmail.com> on 2019/11/01 15:56:08 UTC
Re: Integration tests running on Docker
Hi Daniel,
thanks for following up!
Il giorno mer 30 ott 2019 alle ore 13:05 Daniel Ruggeri
<dr...@primary.net> ha scritto:
>
>
>
> On October 29, 2019 8:12:25 AM CDT, Luca Toscano <to...@gmail.com> wrote:
> >Il giorno mar 29 ott 2019 alle ore 12:23 Joe Orton <jo...@redhat.com>
> >ha scritto:
> >>
> >> On Sun, Oct 27, 2019 at 06:42:58PM +0100, Luca Toscano wrote:
> >> > Some updates:
> >> >
> >> > - We have support for httpd in travis -
> >https://travis-ci.org/apache/httpd
> >>
> >> Nice, thanks Luca & infra!
> >>
> >> > - In order to configure automatic builds, a travis.yaml file is
> >needed
> >> > in the base path of the repository, in every branch that needs to
> >be
> >> > tested IIUC.
> >> > - My idea is currently to add in trunk a travis.yaml initial
> >> > configuration file, that builds a Docker image like [1] and runs
> >the
> >> > perl test suite.
> >> >
> >> > - Building the Docker image in [1] takes quite a while now (between
> >9
> >> > and 13 mins on my laptop, depending also on network bandwidth
> >etc..)
> >> > so it will need more time before it could be as fast as we need,
> >but
> >> > we have to start from somewhere :)
> >> > - In the travis.yaml file we'd need to put config options about
> >what
> >> > Docker image to build and with what parameters. Ideally I'd like to
> >> > store the Docker images in the httpd-framework repository, and
> >>
> >> Do you mean Dockerfiles rather than images here?
> >
> >Correct yes, my bad!
> >
> >>
> >> > reference them in the Travis config of the httpd branches, but not
> >> > sure if it will be possible. Worst case scenario we'll need to add
> >the
> >> > Docker image in each httpd branch that we want to test (possibly in
> >a
> >> > dedicated dir, like "docker" or "testing".
> >>
> >> I expected two goals for testing:
> >>
> >> 1. configure, build, and test using a variety of different configure
> >> options in the standard Ubuntu environment which Travis provides
> >(enable
> >> different MPMs, different module sets, different
> >> --enable-foo/--with-foo, different gcc versions etc etc)
> >
> >This is also what I was trying to understand in these days, namely if
> >travis could build and test directly without Docker.
> >
> >> 2. configure build and test in a variety of OS environments - this
> >would
> >> make sense using container images.
> >>
> >> I don't have much experience testing inside containers from Travis,
> >but
> >> if we can do both (1) and (2) inside containers that might make
> >sense?
> >> Or can we do both separately? If you have a .travis.yml which works
> >> then I'd say go commit it and we can work from there.
> >
> >I have currently a Docker file that works for Debian/Ubuntu (and
> >another one for CentOs), and IIUC the same could also be done for
> >other OSes (so having different Docker files for OSX and Win, since
> >Travis seems to support them). The main problem is that it currently
> >takes minutes to build the Docker images, and only after that tests
> >will be able to run. I guess that this problem will be the same if we
> >run directly tests in Travis, but we can give it a try. Another
> >downside about Travis, in my opinion, is that testing locally seems to
> >be more difficult that building Docker images (but I admit my
> >ignorance so it might be different), this is why I preferred Docker in
> >the first place.
>
> This very closely matches my own testing setup. I *really* like this. Thank you for driving this!
>
> >
> >> Or can we do both separately? If you have a .travis.yml which works
> >> then I'd say go commit it and we can work from there.
> >
> >I currently don't have a working travis yaml, but I'll come up with
> >something during the next days. What I am currently trying to
> >understand is if the Docker files could be stored elsewhere (like in
> >the httpd-framework repo) and downloaded in Travis on demand, to have
> >only one the Travis config in the httpd branches (as opposed to also
> >having Docker files etc..).
> >
> >Let me know your suggestions/thoughts :)
> >
> >Luca
>
> The most I know about Travis is how to spell it, but...
> What if the various Dockerfile-based testing environments were treated as a separate "CI-able" item? That is, what if we monitored for upstream changes then proactively rebuilt and pushed the testing images as "httpdtest-centos-7" and "httpdtest-ubuntu-trusty", etc? That way, the testing image is built/ready so the $work needed on check-in is just to build and test httpd.
>
> A significant downside is that these discreet test "enviroments" will all have their own lifecycles and maintenance burden. I found on my Debian-based Docker test environment that subtle upstream changes caused unexpected breakage. This was pretty disruptive to workflow because I had to get out of my "httpd tester" hat and don my "Debian administrator" hat in the middle of testing whatever I was working on to resolve the breakage. With that in mind, I view the pre-build mechanism as a way to avoid breakages during a testing cycle.
I like the idea, just asked in the task to infra if something like the
following is feasible:
- have a separate github repo for the Dockerfiles (that could even be
the one of the test suite replicated in github in theory)
- after each commit to that repo, an automatic Docker build process is
launched to create base test images in Dockerhub (how to handle
authentication etc..? Open question)
- Add travis config to each httpd branch that we need to test, that
simply pulls the latest images from Dockerhub, checks out the httpd
git commit that is under test, builds it with some ./configure and
runs the perl test suite. This step would be great to be run on
different platforms.
I took a look to Jira and other Apache projects in Github, and didn't
find anything similar to our use case (even if I am pretty sure we are
not the first ones :).
An alternative could be to have something like the following:
- Travis used directly (as Joe mentioned) on multiple platforms to
build httpd after each commit, maybe with different configure options
etc.. Apache Arrow uses the Travis matrix functionality in a similar
way https://github.com/apache/arrow/blob/master/.travis.yml
- add the "make docker platform-x" functionality (as previously
suggested) to the test suite's repo, and have people to run manually
docker tests when needed for multiple platforms.
So basically extending buildbot using Travis, and enrich our testing tooling.
Anyway, I'll wait a bit for infra and keep going with my research, and
as test I'll try to push a Travis config to trunk during the next days
(to have an idea about wait times etc..).
Luca
Re: Integration tests running on Docker
Posted by Stefan Eissing <st...@greenbytes.de>.
Nice work!
Some weeks ago, I played around with travis to get the mod_h2 tests running, but all OS platforms offered included a too old Apache httpd version. Which meant I would need to drag in a recent httpd as well. I got distracted before doing so and this got put back on the useful-things-to-do-list.
Maybe I should fork your fork and add these tests there...
Do you intend to PR this back to httpd one day?
Cheers, Stefan
> Am 04.11.2019 um 17:16 schrieb Joe Orton <jo...@redhat.com>:
>
> Here is a proof of concept -
>
> https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
>
> This runs using the full test suite using a few different configurations
> and also does builds with -Werror and maintainer-mode -
>
> https://travis-ci.org/notroj/httpd/builds/607213820
>
> ...this should be very easy to extend with more configurations.
>
> Can we merge the docker way with this kind of matrix type travis
> configuration?
>
> Regards, Joe
>
Re: Integration tests running on Docker
Posted by Luca Toscano <to...@gmail.com>.
Il giorno gio 7 nov 2019 alle ore 16:49 Yann Ylavic
<yl...@gmail.com> ha scritto:
>
> On Thu, Nov 7, 2019 at 4:47 PM Eric Covener <co...@gmail.com> wrote:
> >
> > > @@ -125,6 +125,7 @@
> > > . non-Unix, single-platform code
> > > . routine APLOGNO() backports
> > > . .gdbinit
> > > + . Travis integration: .travis.yml and test/travis*.sh
> >
> > +1
>
> +1
+1 \o/
Re: Integration tests running on Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Nov 7, 2019 at 4:47 PM Eric Covener <co...@gmail.com> wrote:
>
> > @@ -125,6 +125,7 @@
> > . non-Unix, single-platform code
> > . routine APLOGNO() backports
> > . .gdbinit
> > + . Travis integration: .travis.yml and test/travis*.sh
>
> +1
+1
Re: Integration tests running on Docker
Posted by Eric Covener <co...@gmail.com>.
> @@ -125,6 +125,7 @@
> . non-Unix, single-platform code
> . routine APLOGNO() backports
> . .gdbinit
> + . Travis integration: .travis.yml and test/travis*.sh
+1
Re: Integration tests running on Docker
Posted by Joe Orton <jo...@redhat.com>.
On Thu, Nov 07, 2019 at 10:37:50AM +0000, Pluem, Ruediger, Vodafone Group wrote:
> r1868314
Ah, thank you! Obviously I tested this locally first too but hadn't
updated my test/framework directory... duh.
Should we use RTC or can I get an exception for 2.4.x travis
integration? I have a few more things to merge to trunk then can
backport:
https://github.com/apache/httpd/compare/2.4.x...notroj:travisify-2.4.x
Index: STATUS
===================================================================
--- STATUS (revision 1869499)
+++ STATUS (working copy)
@@ -125,6 +125,7 @@
. non-Unix, single-platform code
. routine APLOGNO() backports
. .gdbinit
+ . Travis integration: .travis.yml and test/travis*.sh
RELEASE SHOWSTOPPERS:
Re: Integration tests running on Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Nov 7, 2019 at 11:31 AM Pluem, Ruediger, Vodafone Group
<ru...@vodafone.com> wrote:
>
> I suspect that $deflated_str_q0 is the result of the request with Accept-Encoding: gzip;q=0
> where mod_deflate actually should not gzip the response because of q=0.
Yeah, brain fart :)
Re: AW: Integration tests running on Docker
Posted by Marion & Christophe JAILLET <ch...@wanadoo.fr>.
Le 07/11/2019 à 11:31, Pluem, Ruediger, Vodafone Group a écrit :
> C2 General
>
>> -----Ursprüngliche Nachricht-----
>> Von: Yann Ylavic <yl...@gmail.com>
>> Gesendet: Donnerstag, 7. November 2019 11:19
>> An: httpd-dev <de...@httpd.apache.org>
>> Betreff: Re: Integration tests running on Docker
>>
>> On Wed, Nov 6, 2019 at 6:54 PM Joe Orton <jo...@redhat.com> wrote:
>>> Has anybody seen t/modules/deflate.t and t/modules/brotli.t fail on
>>> Debian-ish distro? I don't remember seeing this before but maybe it's
>> something
>>> trivial? t/TEST output -
>> It seems to originate from r1868312.
>>
>> I don't really understand this new test in mod_deflate (the one which
>> fails):
>> ok $original_str eq $deflated_str_q0;
>>
>> Why would we expect a plain string to be equal to a deflated one,
>> Christophe?
>> I don't see how the failure depends on debian vs any other system
>> either.
>>
>> Running t/modules/deflate.t on my debian, with this debugging patch:
>> ```
>> Index: t/modules/deflate.t
>> ===================================================================
>> --- t/modules/deflate.t (revision 1869498)
>> +++ t/modules/deflate.t (working copy)
>> @@ -57,6 +57,10 @@ for my $server_deflate_uri (@server_deflate_uris)
>>
>> ok $original_str eq $inflated_str;
>> ok $original_str eq $deflated_str_q0;
>> + if (not $original_str eq $deflated_str_q0) {
>> + print "original: '$original_str'\n";
>> + print "deflated: '$deflated_str_q0'\n";
>> + }
>> my $resp = POST($server_inflate_uri, @inflate_headers,
>> content => "foo123456789012346");
>> if (have_min_apache_version("2.5")) {
>> ```
>>
>> I get (e.g.):
>> ```
>> not ok 42
>> original: '
>> Some dummy content. Some dummy content. Some dummy content. Some dummy
>> content.
>> EOF
>> '
>> deflated: '<some binary (gziped)
> I suspect that $deflated_str_q0 is the result of the request with Accept-Encoding: gzip;q=0
> where mod_deflate actually should not gzip the response because of q=0.
>
Yeap, that's it
> Regards
>
> Rüdiger
AW: Integration tests running on Docker
Posted by "Pluem, Ruediger, Vodafone Group" <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Yann Ylavic <yl...@gmail.com>
> Gesendet: Donnerstag, 7. November 2019 11:19
> An: httpd-dev <de...@httpd.apache.org>
> Betreff: Re: Integration tests running on Docker
>
> On Wed, Nov 6, 2019 at 6:54 PM Joe Orton <jo...@redhat.com> wrote:
> >
> > Has anybody seen t/modules/deflate.t and t/modules/brotli.t fail on
> > Debian-ish distro? I don't remember seeing this before but maybe it's
> something
> > trivial? t/TEST output -
>
> It seems to originate from r1868312.
>
> I don't really understand this new test in mod_deflate (the one which
> fails):
> ok $original_str eq $deflated_str_q0;
>
> Why would we expect a plain string to be equal to a deflated one,
> Christophe?
> I don't see how the failure depends on debian vs any other system
> either.
>
> Running t/modules/deflate.t on my debian, with this debugging patch:
> ```
> Index: t/modules/deflate.t
> ===================================================================
> --- t/modules/deflate.t (revision 1869498)
> +++ t/modules/deflate.t (working copy)
> @@ -57,6 +57,10 @@ for my $server_deflate_uri (@server_deflate_uris)
>
> ok $original_str eq $inflated_str;
> ok $original_str eq $deflated_str_q0;
> + if (not $original_str eq $deflated_str_q0) {
> + print "original: '$original_str'\n";
> + print "deflated: '$deflated_str_q0'\n";
> + }
> my $resp = POST($server_inflate_uri, @inflate_headers,
> content => "foo123456789012346");
> if (have_min_apache_version("2.5")) {
> ```
>
> I get (e.g.):
> ```
> not ok 42
> original: '
> Some dummy content. Some dummy content. Some dummy content. Some dummy
> content.
> EOF
> '
> deflated: '<some binary (gziped)
I suspect that $deflated_str_q0 is the result of the request with Accept-Encoding: gzip;q=0
where mod_deflate actually should not gzip the response because of q=0.
Regards
Rüdiger
AW: Integration tests running on Docker
Posted by "Pluem, Ruediger, Vodafone Group" <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Yann Ylavic <yl...@gmail.com>
> Gesendet: Donnerstag, 7. November 2019 11:32
> An: httpd-dev <de...@httpd.apache.org>
> Betreff: Re: Integration tests running on Docker
>
> On Thu, Nov 7, 2019 at 11:19 AM Yann Ylavic <yl...@gmail.com>
> wrote:
> >
> > Am I missing something?
>
> A brain or something like that :)
>
> Nevermind, I was just confused by the name of $deflated_str_q0, this
> string isn't actually deflated because of q=0.
>
> Since this test passes on trunk, a backport is missing possibly?
r1868314
Regards
Rüdiger
Re: Integration tests running on Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Nov 7, 2019 at 11:31 AM Yann Ylavic <yl...@gmail.com> wrote:
>
> Since this test passes on trunk, a backport is missing possibly?
Yep, r1868313, proposed in STATUS already.
Re: Integration tests running on Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Thu, Nov 7, 2019 at 11:19 AM Yann Ylavic <yl...@gmail.com> wrote:
>
> Am I missing something?
A brain or something like that :)
Nevermind, I was just confused by the name of $deflated_str_q0, this
string isn't actually deflated because of q=0.
Since this test passes on trunk, a backport is missing possibly?
Re: Integration tests running on Docker
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Nov 6, 2019 at 6:54 PM Joe Orton <jo...@redhat.com> wrote:
>
> Has anybody seen t/modules/deflate.t and t/modules/brotli.t fail on
> Debian-ish distro? I don't remember seeing this before but maybe it's something
> trivial? t/TEST output -
It seems to originate from r1868312.
I don't really understand this new test in mod_deflate (the one which fails):
ok $original_str eq $deflated_str_q0;
Why would we expect a plain string to be equal to a deflated one, Christophe?
I don't see how the failure depends on debian vs any other system either.
Running t/modules/deflate.t on my debian, with this debugging patch:
```
Index: t/modules/deflate.t
===================================================================
--- t/modules/deflate.t (revision 1869498)
+++ t/modules/deflate.t (working copy)
@@ -57,6 +57,10 @@ for my $server_deflate_uri (@server_deflate_uris)
ok $original_str eq $inflated_str;
ok $original_str eq $deflated_str_q0;
+ if (not $original_str eq $deflated_str_q0) {
+ print "original: '$original_str'\n";
+ print "deflated: '$deflated_str_q0'\n";
+ }
my $resp = POST($server_inflate_uri, @inflate_headers,
content => "foo123456789012346");
if (have_min_apache_version("2.5")) {
```
I get (e.g.):
```
not ok 42
original: '
Some dummy content. Some dummy content. Some dummy content. Some dummy content.
EOF
'
deflated: '<some binary (gziped)
stuff, expected I guess>'
# Failed test 42 in t/modules/deflate.t at line 59 fail #11
```
Am I missing something?
Regards,
Yann.
Re: Integration tests running on Docker
Posted by Joe Orton <jo...@redhat.com>.
Has anybody seen t/modules/deflate.t and t/modules/brotli.t fail on
Debian-ish distro? I don't remember seeing this before but maybe it's something
trivial? t/TEST output -
https://travis-ci.org/notroj/httpd/jobs/608306769#L3244
(I'm trying to get Travis working in 2.4.x as well.)
Regards, Joe
AW: Integration tests running on Docker
Posted by "Pluem, Ruediger, Vodafone Group" <ru...@vodafone.com>.
C2 General
> -----Ursprüngliche Nachricht-----
> Von: Joe Orton <jo...@redhat.com>
> Gesendet: Mittwoch, 6. November 2019 10:53
> An: Apache HTTP Server Development List <de...@httpd.apache.org>
> Betreff: Re: Integration tests running on Docker
>
> On Tue, Nov 05, 2019 at 10:06:23AM +0100, Luca Toscano wrote:
> > Basically what Joe did, but following what the Apache Arrow Project
> > did. There is a little bit of repetition, but in theory with this
> > config people are free to add tests for other OSes (osx, windows) and
> > I will probably be able to add custom ones with Docker to see the
> > difference in execution timings etc.. (Apache Arrow's test do use
> > docker, I didn't see it a first).
>
> I haven't tried to follow all of what the Arrow .yml is doing (too much
> Java ;) but moving some of the logic into the repo itself out of
> .travis.yml seems like a good idea. e.g. moving all of script: into
> something like ./test/travis_script_linux.sh or something for the Linux
> based tests, does that make sense?
>
> This is working now -
>
> https://travis-ci.org/apache/httpd/builds
>
> and I guess it will work for forks and PRs too, haven't tried yet.
>
> Anybody feel free to extend, play, hack, whatever with the travis.yml, I
> am no expert so may be doing something non-standard/wrong, feel free to
> correct/improve ;)
>
> The jobs are quite quick but the queue time for the "apache" repos is
> long (10+ minutes) before they get started. I guess it is doing
> notifications to committers directly, but we won't find out until a job
> fails? Maybe let's keep experimenting with this a bit longer and then
> start sending mail to dev@ for state changes (exactly as buildbot does)
> when it seems stable?
+1. Thanks for taking care.
Regards
Rüdiger
Re: Integration tests running on Docker
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Nov 05, 2019 at 10:06:23AM +0100, Luca Toscano wrote:
> Basically what Joe did, but following what the Apache Arrow Project
> did. There is a little bit of repetition, but in theory with this
> config people are free to add tests for other OSes (osx, windows) and
> I will probably be able to add custom ones with Docker to see the
> difference in execution timings etc.. (Apache Arrow's test do use
> docker, I didn't see it a first).
I haven't tried to follow all of what the Arrow .yml is doing (too much
Java ;) but moving some of the logic into the repo itself out of
.travis.yml seems like a good idea. e.g. moving all of script: into
something like ./test/travis_script_linux.sh or something for the Linux
based tests, does that make sense?
This is working now -
https://travis-ci.org/apache/httpd/builds
and I guess it will work for forks and PRs too, haven't tried yet.
Anybody feel free to extend, play, hack, whatever with the travis.yml, I
am no expert so may be doing something non-standard/wrong, feel free to
correct/improve ;)
The jobs are quite quick but the queue time for the "apache" repos is
long (10+ minutes) before they get started. I guess it is doing
notifications to committers directly, but we won't find out until a job
fails? Maybe let's keep experimenting with this a bit longer and then
start sending mail to dev@ for state changes (exactly as buildbot does)
when it seems stable?
Regards, Joe
Re: Integration tests running on Docker
Posted by Luca Toscano <to...@gmail.com>.
Il giorno mer 6 nov 2019 alle ore 12:15 Joe Orton <jo...@redhat.com>
ha scritto:
>
> On Wed, Nov 06, 2019 at 10:54:56AM +0100, Luca Toscano wrote:
> > I just noticed https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems,
> > so the 'addons' will be used by travis only if the os can use them (I
> > was afraid of having windows and apt combined). The only thing that I
> > would add to travis.yml are the ifs to separate
> > before_install/script/etc.. for every os, I think that we should add
> > windows as soon as possible.. Do we have a list of build commands to
> > test in Travis?
>
> I have no idea how to build on anything but Linux :) Someone could do
> osx too, travis supports that...
Me too, I am not a big Windows user, but I am pretty sure that
somebody else in the dev community have some info! Will wait :)
> So it works for PRs and successfully failed for a bogus commit -
>
> https://travis-ci.org/apache/httpd/pull_requests
>
> This one is sensible?
>
> https://github.com/apache/httpd/pull/71/commits/4a46327813e87f3d662e65361e6d9240ce855db5
+1 looks good, we can either keep the ifs in travis config or go for
specific files, I like both approaches!
> I don't know if anything other than Linux can actually be tested like
> this tbh. (Also trying to context switch between svn & git doing this
> stuff is going to melt my brain!)
We could start with a simple build on Windows, I think it would be
sufficient for the moment.. Osx should be easier, will try to add
something as well. The current format of travis.yaml is simple enough
that people with experience can add their tests, but surely some
documentation is needed!
Luca
Re: Integration tests running on Docker
Posted by Joe Orton <jo...@redhat.com>.
On Wed, Nov 06, 2019 at 10:54:56AM +0100, Luca Toscano wrote:
> I just noticed https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems,
> so the 'addons' will be used by travis only if the os can use them (I
> was afraid of having windows and apt combined). The only thing that I
> would add to travis.yml are the ifs to separate
> before_install/script/etc.. for every os, I think that we should add
> windows as soon as possible.. Do we have a list of build commands to
> test in Travis?
I have no idea how to build on anything but Linux :) Someone could do
osx too, travis supports that...
So it works for PRs and successfully failed for a bogus commit -
https://travis-ci.org/apache/httpd/pull_requests
This one is sensible?
https://github.com/apache/httpd/pull/71/commits/4a46327813e87f3d662e65361e6d9240ce855db5
I don't know if anything other than Linux can actually be tested like
this tbh. (Also trying to context switch between svn & git doing this
stuff is going to melt my brain!)
Re: Integration tests running on Docker
Posted by Luca Toscano <to...@gmail.com>.
Il giorno mar 5 nov 2019 alle ore 10:06 Luca Toscano
<to...@gmail.com> ha scritto:
>
> Il giorno lun 4 nov 2019 alle ore 19:33 Luca Toscano
> <to...@gmail.com> ha scritto:
> >
> > Hi Joe,
> >
> > Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton <jo...@redhat.com>
> > ha scritto:
> > >
> > > Here is a proof of concept -
> > >
> > > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
> > >
> > > This runs using the full test suite using a few different configurations
> > > and also does builds with -Werror and maintainer-mode -
> > >
> > > https://travis-ci.org/notroj/httpd/builds/607213820
> > >
> > > ...this should be very easy to extend with more configurations.
> > >
> > > Can we merge the docker way with this kind of matrix type travis
> > > configuration?
> > >
> >
> > The proof of concept looks great, I wanted to test something similar
> > especially to get timings. It looks like a single test takes from 2 to
> > 5 minutes maximum, that is really impressive, I thought it would have
> > been more. With Docker in theory we could use the os matrix, but the
> > documentation doesn't really suggest any best practice. My idea was to
> > pull different docker images (previously built and uploaded to Docker
> > Hub), and depending on the os in the matrix run the correct docker
> > command in the "script" section (to get an idea
> > https://github.com/apache/arrow/blob/master/.travis.yml, even if they
> > don't use docker).
> >
> > While I research though it might be good to follow your solution and
> > just have a simple Travis file for say ubuntu and windows. Even if it
> > takes 30 mins to complete, it would be a good feedback for anybody
> > committing code to trunk/2.4.x. And it could be possible that this
> > solution is good enough for our use cases, only testing will tell us!
> > So feel free to commit your Travis config to trunk, maybe following
> > the Apache Arrow example? If so others, more experienced with Windows,
> > might be able to chime in and add the missing configuration.
>
> This is my idea:
>
> https://github.com/elukey/httpd/blob/trunk/.travis.yml
>
> https://travis-ci.org/elukey/httpd/builds/607536348
>
> Basically what Joe did, but following what the Apache Arrow Project
> did. There is a little bit of repetition, but in theory with this
> config people are free to add tests for other OSes (osx, windows) and
> I will probably be able to add custom ones with Docker to see the
> difference in execution timings etc.. (Apache Arrow's test do use
> docker, I didn't see it a first).
>
I just noticed https://docs.travis-ci.com/user/installing-dependencies/#installing-dependencies-on-multiple-operating-systems,
so the 'addons' will be used by travis only if the os can use them (I
was afraid of having windows and apt combined). The only thing that I
would add to travis.yml are the ifs to separate
before_install/script/etc.. for every os, I think that we should add
windows as soon as possible.. Do we have a list of build commands to
test in Travis?
Luca
Re: Integration tests running on Docker
Posted by Luca Toscano <to...@gmail.com>.
Il giorno lun 4 nov 2019 alle ore 19:33 Luca Toscano
<to...@gmail.com> ha scritto:
>
> Hi Joe,
>
> Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton <jo...@redhat.com>
> ha scritto:
> >
> > Here is a proof of concept -
> >
> > https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
> >
> > This runs using the full test suite using a few different configurations
> > and also does builds with -Werror and maintainer-mode -
> >
> > https://travis-ci.org/notroj/httpd/builds/607213820
> >
> > ...this should be very easy to extend with more configurations.
> >
> > Can we merge the docker way with this kind of matrix type travis
> > configuration?
> >
>
> The proof of concept looks great, I wanted to test something similar
> especially to get timings. It looks like a single test takes from 2 to
> 5 minutes maximum, that is really impressive, I thought it would have
> been more. With Docker in theory we could use the os matrix, but the
> documentation doesn't really suggest any best practice. My idea was to
> pull different docker images (previously built and uploaded to Docker
> Hub), and depending on the os in the matrix run the correct docker
> command in the "script" section (to get an idea
> https://github.com/apache/arrow/blob/master/.travis.yml, even if they
> don't use docker).
>
> While I research though it might be good to follow your solution and
> just have a simple Travis file for say ubuntu and windows. Even if it
> takes 30 mins to complete, it would be a good feedback for anybody
> committing code to trunk/2.4.x. And it could be possible that this
> solution is good enough for our use cases, only testing will tell us!
> So feel free to commit your Travis config to trunk, maybe following
> the Apache Arrow example? If so others, more experienced with Windows,
> might be able to chime in and add the missing configuration.
This is my idea:
https://github.com/elukey/httpd/blob/trunk/.travis.yml
https://travis-ci.org/elukey/httpd/builds/607536348
Basically what Joe did, but following what the Apache Arrow Project
did. There is a little bit of repetition, but in theory with this
config people are free to add tests for other OSes (osx, windows) and
I will probably be able to add custom ones with Docker to see the
difference in execution timings etc.. (Apache Arrow's test do use
docker, I didn't see it a first).
Luca
Re: Integration tests running on Docker
Posted by Luca Toscano <to...@gmail.com>.
Hi Joe,
Il giorno lun 4 nov 2019 alle ore 17:16 Joe Orton <jo...@redhat.com>
ha scritto:
>
> Here is a proof of concept -
>
> https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
>
> This runs using the full test suite using a few different configurations
> and also does builds with -Werror and maintainer-mode -
>
> https://travis-ci.org/notroj/httpd/builds/607213820
>
> ...this should be very easy to extend with more configurations.
>
> Can we merge the docker way with this kind of matrix type travis
> configuration?
>
The proof of concept looks great, I wanted to test something similar
especially to get timings. It looks like a single test takes from 2 to
5 minutes maximum, that is really impressive, I thought it would have
been more. With Docker in theory we could use the os matrix, but the
documentation doesn't really suggest any best practice. My idea was to
pull different docker images (previously built and uploaded to Docker
Hub), and depending on the os in the matrix run the correct docker
command in the "script" section (to get an idea
https://github.com/apache/arrow/blob/master/.travis.yml, even if they
don't use docker).
While I research though it might be good to follow your solution and
just have a simple Travis file for say ubuntu and windows. Even if it
takes 30 mins to complete, it would be a good feedback for anybody
committing code to trunk/2.4.x. And it could be possible that this
solution is good enough for our use cases, only testing will tell us!
So feel free to commit your Travis config to trunk, maybe following
the Apache Arrow example? If so others, more experienced with Windows,
might be able to chime in and add the missing configuration.
Thanks!
Luca
Re: Integration tests running on Docker
Posted by Joe Orton <jo...@redhat.com>.
Here is a proof of concept -
https://github.com/notroj/httpd/blob/a73adfc8f1c01fbb6a3d493582f9c49c7c942756/.travis.yml
This runs using the full test suite using a few different configurations
and also does builds with -Werror and maintainer-mode -
https://travis-ci.org/notroj/httpd/builds/607213820
...this should be very easy to extend with more configurations.
Can we merge the docker way with this kind of matrix type travis
configuration?
Regards, Joe