new VM whimsy-test site and VM

[Sam Ruby <ru...@intertwingly.net>]

(bcc: infra@)

TL;DR: the goal of this VM is to make it so that everybody on the Whimsy 
PMC can deploy both changes and new applications.  This is a progress 
report.

---

For starters, here is the new website on a new VM:

https://whimsy-test.apache.org/

The new VM is based on Ubuntu 14.04 (the current Whimsy is based on 
Ubuntu 12.04), and is more completely managed by Puppet.  The puppet 
definition is contained in the following two files:

Content:

https://github.com/apache/whimsy

VM Definition:

https://github.com/apache/infrastructure-puppet/blob/deployment/data/nodes/whimsy-vm2.apache.org.yaml

https://github.com/apache/infrastructure-puppet/blob/deployment/modules/whimsy_server/manifests/init.pp

Instructions:

https://github.com/rubys/puppet-kitchen#readme

https://github.com/apache/infrastructure-puppet/blob/deployment/modules/vhosts_whimsy/README.md

Workflow:

https://cwiki.apache.org/confluence/display/INFRA/Git+workflow+for+infrastructure-puppet+repo

---

Applications consist of static data, CGI scripts, Rack applications, and 
a single Puma application (the board agenda, which has some unique 
response time requirements), and cron jobs.  Current status:

(*) static content is working.  Changes pushed to GitHub master will be 
automatically deployed every 30 minutes.  Note that this includes the 
contents of scripts and applications too.

(*) CGI scripts are working, but dependencies (gems, svn checkouts) have 
not been installed.  A simple CGI:

https://github.com/apache/whimsy/blob/master/www/test.cgi
https://whimsy-test.apache.org/test.cgi

Many CGI scripts will require user authentication.  This is done by 
adding a single line to the deployment data identifying the location of 
the script:

https://github.com/apache/infrastructure-puppet/blob/deployment/data/nodes/whimsy-vm2.apache.org.yaml#L65

Note that the LDAP module does not currently handle boolean conditions 
(example: members or officers).  The way to handle this is to do 
authentication in two passes.  The first pass will be done by the Apache 
web server, and verify that the user is a part of the most inclusive 
group (typically: committers).  The CGI scripts that need to do more 
will need to

(*) Rack applications are working and run under Passenger under Apache 
httpd.  Again, dependencies are not yet ready.  In addition to simply 
checking the application, one line per passenger application needs to be 
added to the deployment data:

https://github.com/apache/infrastructure-puppet/blob/deployment/data/nodes/whimsy-vm2.apache.org.yaml#L60

A simple rack application (two empty directories, and a one line file):

https://github.com/apache/whimsy/tree/master/www/racktest
https://whimsy-test.apache.org/racktest

Authentication requirements will also need to be two phase, like with 
CGI above; but more common conditions can be handled at the "Rack" level 
instead of at the application level making use of Rack middleware such as:

https://github.com/apache/whimsy/blob/master/lib/whimsy/asf/rack.rb#L56

(*) I've not started work on deploying the Puma application, but have 
verified that there is a Puppet module that handle do the necessary 
deployment.

(*) I've not started work on cron jobs.  The definitions for these are 
likely to be placed in the infrastructure-puppet repository, with the 
code that are executed by the cron jobs placed in the whimsy repository.

---

The biggest todos are:

1) get a handle on how to test LDAP changes locally.  LDAP is installed 
on whimsy-vm2, but not on local puppet-kitchen-vagrant repositories.

2) define a strategy for maintaining checkout data (mostly SVN) which is 
needed by multiple applications.  For production, that will mean 
deployment on things like credentials for the whimsysvn user.  For local 
testing that likely will mean mounting local directories by the VM.

3) define a strategy for maintaining dependencies.  Whimsy (as a site) 
hosts multiple independent applications, each of which may have 
different requirements.  I'd like to make it so that not every 
application has to be upgraded at the same time when a new version of a 
dependency comes out.  This likely will involve the use of bundler 
(https://forge.puppetlabs.com/ploperations/bundler).

4) deploy puma (https://forge.puppetlabs.com/deversus/puma)

5) set up cron jobs.

---

A big shout out to to the infrastructure team for helping me get this 
far.  It has been a team effort with help from (at a minimum) Chris, 
Daniel, David, Geoff, and Tony.  With the investment in Puppet 
automation, replacing a dependency, the OS, or even the VM entirely 
should be relatively painless.

- Sam Ruby