You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jacob Champion <ch...@gmail.com> on 2017/01/30 20:02:35 UTC
Autobuild Progress (was Re: Automated tests)
On 01/02/2017 07:53 AM, Daniel Shahaf wrote:
> Setting this up isn't a lot more complicated than filing an INFRA ticket
> with a build script, a list of build dependencies, and a list of
> branches to build, and deciding how build failures would be notified.
To follow up on this, we now have an operational (if not yet fully
functional) buildbot for trunk:
https://ci.apache.org/builders/httpd-trunk
There's a lot of work yet to do, but for now we have an Ubuntu machine
that can be manually triggered to run an incremental build on trunk.
Here's my list of TODOs:
- run per-commit incremental builds
- run nightly clean builds
- run a nightly test suite
- set up 2.4.x in addition to trunk
- set up Windows in addition to Ubuntu
== Details ==
The bot is building against Ubuntu-packaged dependencies, which requires
a new apr-config option for buildconf (run `./buildconf --help` on the
latest trunk for info). This leaves out a few modules that need some
bleeding-edge dependencies:
- mod_brotli (needs the unreleased libbrotli)
- mod_crypto (needs APR 1.6)
- mod[_proxy]_http2 (needs libnghttp2)
- mod_lua (needs our configure script to recognize Lua 5.3)
So to run a full test suite, eventually we'll need to build those
dependencies too. I figure this is a good start for now.
The following modules aren't built because of platform-specific stuff:
- mod_socache_dc (distcache)
- mod_journald, mod_systemd
- mod_privileges (sys/priv.h)
- mpm_os2
- mpm_winnt
If you'd like to poke around, our buildbot configuration file is in the
infra repository and is, I believe, open to all our committers:
https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/httpd.conf
--Jacob
Re: Autobuild Progress (was Re: Automated tests)
Posted by Jacob Champion <ch...@gmail.com>.
On 01/30/2017 06:23 PM, Daniel Shahaf wrote:
> You could also configure buildbot to detect any new compiler warnings.
> That'll cause buildbot to report a FAILURE (red) or a WARNING (yellow),
> instead of a SUCCESS (green), if the build succeeded but a compiler
> warning has been issued.
Good suggestion. I *think* the eventual goal is to move to -Werror for
maintainer-mode builds, which would remove buildbot from the equation
entirely (and force committers to fix their stuff before committing :P),
but there's currently a bug in trunk's configure script that keeps it
from being enabled as intended.
I'll keep this in mind in case enabling -Werror proves to have other
issues (or if we come across a platform without a reliable -Werror). Thanks!
--Jacob
Re: Autobuild Progress (was Re: Automated tests)
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Jacob Champion wrote on Mon, Jan 30, 2017 at 12:02:35 -0800:
> On 01/02/2017 07:53 AM, Daniel Shahaf wrote:
> >Setting this up isn't a lot more complicated than filing an INFRA ticket
> >with a build script, a list of build dependencies, and a list of
> >branches to build, and deciding how build failures would be notified.
>
> To follow up on this, we now have an operational (if not yet fully
> functional) buildbot for trunk:
>
> https://ci.apache.org/builders/httpd-trunk
>
Nice!
> There's a lot of work yet to do, but for now we have an Ubuntu machine that
> can be manually triggered to run an incremental build on trunk.
>
> Here's my list of TODOs:
> - run per-commit incremental builds
> - run nightly clean builds
> - run a nightly test suite
> - set up 2.4.x in addition to trunk
> - set up Windows in addition to Ubuntu
You could also configure buildbot to detect any new compiler warnings.
That'll cause buildbot to report a FAILURE (red) or a WARNING (yellow),
instead of a SUCCESS (green), if the build succeeded but a compiler
warning has been issued.
(Currently, the build passes the -Wall flag and has no warnings.)
You can crib the setup from the 'svn-warnings' builder in
subversion.conf in the same directory.
Cheers,
Daniel
Re: Autobuild Progress
Posted by Eric Covener <co...@gmail.com>.
On Tue, Mar 7, 2017 at 3:24 PM, Jacob Champion <ch...@gmail.com> wrote:
> Getting 2.4.x going will require some backports, so I'm planning to look
> into running the test suite against trunk (probably not next week but the
> week after). Unless there's anyone who *really* wants 2.4.x autobuilding
> ASAP. Thoughts?
I think the tests are probably a higher priority.
--
Eric Covener
covener@gmail.com
Re: Autobuild Progress
Posted by Jacob Champion <ch...@gmail.com>.
On 02/02/2017 12:22 PM, Jacob Champion wrote:
> Every commit you make to trunk (well, group of commits, within
> fifteen seconds of each other) is run through an incremental build,
> which takes about ten seconds. Every eight hours, the build tree is
> clobbered, resync'd, and built from scratch, which takes about a
> minute.
>
> Reporting is still IRC-only. I want to keep an eye on the bots for bit
> before unleashing them upon the mailing list.
Since we're having a parallel discussion about the test suite, it seems
like a good time to update you on this piece as well.
https://ci.apache.org/builders/httpd-trunk
The builder has been happily chugging away for over a month. It caught
two bad trunk commits (both of which, IIRC, were fixed independently
within ten minutes). There's been only one spurious failure, related to
SVN complaining about DNS hostnames in the middle of a sync. (This
happened around the same time that apache.org hosts were undergoing some
sort of certificate strangeness, so I'm not really worried about it yet.)
So I think it's safe to start sending build reports to the mailing list.
We have the following wishlist items left:
- run a nightly test suite
- set up 2.4.x in addition to trunk
- set up Windows in addition to Ubuntu
Getting 2.4.x going will require some backports, so I'm planning to look
into running the test suite against trunk (probably not next week but
the week after). Unless there's anyone who *really* wants 2.4.x
autobuilding ASAP. Thoughts?
--Jacob
Re: Autobuild Progress (was Re: Automated tests)
Posted by Jacob Champion <ch...@gmail.com>.
On 01/30/2017 12:02 PM, Jacob Champion wrote:
> - run per-commit incremental builds
> - run nightly clean builds
These two are implemented. Every commit you make to trunk (well, group
of commits, within fifteen seconds of each other) is run through an
incremental build, which takes about ten seconds. Every eight hours, the
build tree is clobbered, resync'd, and built from scratch, which takes
about a minute.
Reporting is still IRC-only. I want to keep an eye on the bots for bit
before unleashing them upon the mailing list.
--Jacob
Re: Autobuild Progress (was Re: Automated tests)
Posted by Daniel Ruggeri <DR...@primary.net>.
On 1/31/2017 4:30 PM, Jacob Champion wrote:
> On 01/30/2017 05:39 PM, Daniel Ruggeri wrote:
>> I'm tremendously inspired by this work. What are your thoughts on the
>> idea of having a series of docker container builds that compile and run
>> the test suite on various distributions? I'll volunteer to give this a
>> whack since it's something that's been in the back of my mind for a long
>> while...
>
> I think that would be awesome. The cheaper we can make new test
> distributions, the easier we can test all sorts of different
> configurations (which, given how many knobs and buttons we expose, is
> important).
>
> I don't know how much of Infra's current Puppet/Buildbot framework is
> Docker-friendly, but if there's currently no cheap virtualization
> solution there for build slaves, then anything we added would
> potentially be useful for other ASF projects as well. Definitely
> something to start a conversation over.
>
Yes, definitely. Thinking more about this, even adding something
heavyweight like a type 2 hypervisor could potentially provide value so
long as the VM image is stripped down enough and we don't leave junk
behind on the slave. I'm not concerned about Puppet and buildbot
integration since Puppet is a great way to manage the configuration of
the slave (assuming that's what it's used for) which makes it easy to
have docker, virtualbox, vagrant or whatever installed and configured.
As far as buildbot, I'm sure it will support the execution of a script
which is all that's needed. My latest work with the
RemoteIPProxyProtocol stuff has me compiling httpd on my build machine
and standing up a docker container with haproxy inside. Hitting the
resulting build under various circumstances with wget scratches the
itch. I've got this distilled down into only four files (Dockerfile,
haproxy.cfg, setup script and test script). This is nice because...
well... I just don't want to install haproxy on my build box for this
In any event, I've started the conversation with builds@a.o to see
what's doable. Can crosspost or just return with feedback when I hear.
> (Side thought: since raw speed is typically one of the top priorities
> for a CI test platform, we'd have to carefully consider which items we
> tested by spinning up containers and which we ran directly on a
> physical machine. Though I don't know how fast Docker has gotten with
> all of the fancy virtualization improvements.)
Amen to that. Docker's quite fast since lxc and all the stuff around it
are very lightweight. The slowest parts are pulling the base image and
setting it up (installing compilers, the test framework, tools, etc).
This can be sped up greatly by building the image and publishing it back
to a (or "the") registry or keeping it local on the machine, but we'd
then have to maintain images which I'm not a fan of.
>
>> I think with the work you've done and plan to do, a step like above to
>> increase our ability to test against many distributions all at once (and
>> cheaply) and also making the test framework more approachable, we could
>> seriously increase our confidence when pulling the trigger on a release
>> or accepting a backport.
>
> +1. It'll take some doing (mostly focused on the coverage of the test
> suite itself), but we can get there.
>
>> I'm also a big fan of backports requiring tests, but am honestly
>> intimidated by the testing framework...
>
> What would make it less intimidating for you? (I agree with you, but
> I'm hoping to get your take without biasing it with my already-strong
> opinions. :D)
Opinions here... so take them with a grain of salt.
* The immediate barrier to entry is doco. From the test landing page,
you are greeted with a list of what's in the test project and links to
the components. Of the links there (our super important one is Perl
Framework), only the flood link leads to a useful getting started guide.
This may be lazy and kinda preachy, but not having good developer info
easily accessible is a Bad Thing(tm) since it's a surefire way to scare
off those potentially interested in participating in a project.
* It's also intimidating when a developer realizes they need to learn a
new skill set to create tests. Writing tests for Perl's testing
framework feels archaic, and I'm not sure is a skill many potential
contributors would possess unless they have developed Perl modules for
distribution. I understand the history of the suite so I _get_ why it's
this way... it's just that it is likely a turn-off. Disclaimer: I'm not
saying Perl has a bad testing framework. I have yet to find a testing
framework I'm a big fan of since they all have their idiosyncrasies. No
holy wars, please :-)
* Another barrier that I think is very much worth pointing out is that
several Perl modules must be installed. I have some history fighting
with Crypt::SSLeay to do what I want because it can be rather finicky.
For example, if your system libssl is something ancient like 0.9.8 but
you compiled httpd with 1.0.2, you'll have a bad time (unless you do
some acrobatics to compile/install the module by hand) trying to speak
modern crypto algorithms.
* The setup activities for the test framework also imply root access.
It's definitely possible to install CPAN modules in a local directory,
but that again also requires acrobatics. Some folks don't have root or
just don't want to install system-wide stuff for just one project. Other
testing frameworks use the same runtime to test as the code does to
execute (JUnit as an example).
* It also feels weird that the test project is separate and that I can't
run `make test'. This is a spinal reflex for sysadmins after compiling
software. Not really an 'intimidation' thing... just... weird.
These are reasons why I love the idea of using Docker for building and
testing. In a cleanroom pseudo-installation, you have complete control
of the environment and can manipulate it/throw it away. The immutability
also ensures you build and test from a known state. It also helps that
with a few changes to a Dockerfile I can switch from building and
testing on Debian to Ubuntu in minutes.
>
> --Jacob
--
Daniel Ruggeri
Re: Autobuild Progress (was Re: Automated tests)
Posted by Jacob Champion <ch...@gmail.com>.
On 01/30/2017 05:39 PM, Daniel Ruggeri wrote:
> I'm tremendously inspired by this work. What are your thoughts on the
> idea of having a series of docker container builds that compile and run
> the test suite on various distributions? I'll volunteer to give this a
> whack since it's something that's been in the back of my mind for a long
> while...
I think that would be awesome. The cheaper we can make new test
distributions, the easier we can test all sorts of different
configurations (which, given how many knobs and buttons we expose, is
important).
I don't know how much of Infra's current Puppet/Buildbot framework is
Docker-friendly, but if there's currently no cheap virtualization
solution there for build slaves, then anything we added would
potentially be useful for other ASF projects as well. Definitely
something to start a conversation over.
(Side thought: since raw speed is typically one of the top priorities
for a CI test platform, we'd have to carefully consider which items we
tested by spinning up containers and which we ran directly on a physical
machine. Though I don't know how fast Docker has gotten with all of the
fancy virtualization improvements.)
> I think with the work you've done and plan to do, a step like above to
> increase our ability to test against many distributions all at once (and
> cheaply) and also making the test framework more approachable, we could
> seriously increase our confidence when pulling the trigger on a release
> or accepting a backport.
+1. It'll take some doing (mostly focused on the coverage of the test
suite itself), but we can get there.
> I'm also a big fan of backports requiring tests, but am honestly
> intimidated by the testing framework...
What would make it less intimidating for you? (I agree with you, but I'm
hoping to get your take without biasing it with my already-strong
opinions. :D)
--Jacob
Re: Autobuild Progress (was Re: Automated tests)
Posted by Daniel Ruggeri <DR...@primary.net>.
I'm tremendously inspired by this work. What are your thoughts on the
idea of having a series of docker container builds that compile and run
the test suite on various distributions? I'll volunteer to give this a
whack since it's something that's been in the back of my mind for a long
while...
I think with the work you've done and plan to do, a step like above to
increase our ability to test against many distributions all at once (and
cheaply) and also making the test framework more approachable, we could
seriously increase our confidence when pulling the trigger on a release
or accepting a backport.
P.S.
I'm also a big fan of backports requiring tests, but am honestly
intimidated by the testing framework...
--
Daniel Ruggeri
On 1/30/2017 2:02 PM, Jacob Champion wrote:
> On 01/02/2017 07:53 AM, Daniel Shahaf wrote:
>> Setting this up isn't a lot more complicated than filing an INFRA ticket
>> with a build script, a list of build dependencies, and a list of
>> branches to build, and deciding how build failures would be notified.
>
> To follow up on this, we now have an operational (if not yet fully
> functional) buildbot for trunk:
>
> https://ci.apache.org/builders/httpd-trunk
>
> There's a lot of work yet to do, but for now we have an Ubuntu machine
> that can be manually triggered to run an incremental build on trunk.
>
> Here's my list of TODOs:
> - run per-commit incremental builds
> - run nightly clean builds
> - run a nightly test suite
> - set up 2.4.x in addition to trunk
> - set up Windows in addition to Ubuntu
>
> == Details ==
>
> The bot is building against Ubuntu-packaged dependencies, which
> requires a new apr-config option for buildconf (run `./buildconf
> --help` on the latest trunk for info). This leaves out a few modules
> that need some bleeding-edge dependencies:
>
> - mod_brotli (needs the unreleased libbrotli)
> - mod_crypto (needs APR 1.6)
> - mod[_proxy]_http2 (needs libnghttp2)
> - mod_lua (needs our configure script to recognize Lua 5.3)
>
> So to run a full test suite, eventually we'll need to build those
> dependencies too. I figure this is a good start for now.
>
> The following modules aren't built because of platform-specific stuff:
>
> - mod_socache_dc (distcache)
> - mod_journald, mod_systemd
> - mod_privileges (sys/priv.h)
> - mpm_os2
> - mpm_winnt
>
> If you'd like to poke around, our buildbot configuration file is in
> the infra repository and is, I believe, open to all our committers:
>
>
> https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/httpd.conf
>
>
> --Jacob