Deployment update: secmail

[Sam Ruby <ru...@intertwingly.net>]

tl;dr: mail will be processed more reliably and with less latency.

----

Not yet connected to SVN, and only accessible by the secretarial 
assistant team[1]:

   https://whimsy-test.apache.org/secmail/

Highlights: auto-deployment, and secretary mail subscription.  Not being 
connected to svn means "file" doesn't work.  There probably are other 
bugs (e.g., I note that signature verification blows up).  These will be 
fixed over time.

Regarding auto-deployment. Commit a change to 
https://github.com/apache/whimsy/tree/master/www/secmail and it will be 
deployed the next time Puppet runs.  Puppet is scheduled to run every 30 
minutes.

---

Now for the bigger news, regarding secretary mail subscription:


Old process:

1) minotaur is subscribed to secretary@ (more precisely: listed in the 
alias for secretary@).

2) minotaur files emails received to the appropriate mbox file

3) every 30 minutes (used to be 10, but there were problems), a script 
would look for updates to the mbox file, parse the new emails, and store 
the results in svn in the documents/received folder.

4) every time the secretary loads the main workbench page svn is updated 
(there also is a cron job to do this every 60 minutes).

5) documents placed into svn would then be deleted or moved as they are 
processed by the secretary.


New process:

1) whimsy-vm2 is subscribed to secretary@ (i.e., listed in the alias)

2) whimsy-vm2 parses emails as they are received.

3) selected documents will be imported into svn as they are processed by 
the secretary


Advantages: less latency and less moving parts.  If there was a hiccup 
with LDAP or SVN or a particularly malformed spam email, the flow could 
get interrupted and require manual intervention.  Also, minotaur and 
access to mail archives are scheduled to be replaced, this new flow will 
not be affected by those changes.

Mail servers automatically retry when the target site is down - I sent 
test emails before this was set up and they were processed the next day 
when I got it working.

Should there be a problem with LDAP or SVN, it will only affect the 
filing step; the secretary can simply retry that same step later.

Mails are processed individually as they are received.  While my new 
parsing logic can handle every email received by the secretary since 
October of 2007, should something new come in that breaks the parser, 
the most it can affect is that one email.

Worst case recovery, if an email needs to be reprocessed: resend or 
forward that email to secretary@whimsy-vm2.apache.org (or to 
secretary@apache.org).


- Sam Ruby

[1] https://whimsy.apache.org/roster/group/asf-secretary

Re: Deployment update: secmail

[Sam Ruby <ru...@intertwingly.net>]

Latest changes: the index page now updates live. And a button to
request uploading of keys.

If an email comes in while you are on that page, it will show up
within a second.  If you have delete an email in a separate tab, it
will disappear in both.

You can see the new button here:

https://whimsy-test.apache.org/secmail/201601/c69741a16b/

Click on the pdf link.

More importantly, I want to walk you through the implementation:

https://github.com/apache/whimsy/commit/ca23db87bfaa3824c4930a7729093a89cae481db

In the render method of CheckSignature, you will see an if statement
that checks for text in the alert.  Some people argue against
combining logic and views, but React.js positively revels in it.
Anyway, if the text is found, a button is rendered, with specific text
and the button is associated with a method that is called when the
button is clicked.

That method (request_upload) constructs an email and sets the window
location in order to launch your email client.

I'm guessing that you will want to tailor the message, and may be
interested in adding other buttons (e.g. for invalid signatures?).

It is my hope that I can hand off this work to you and focus on
getting filing of ICLAs working.

- Sam Ruby