Fiabilization of Processing in James

[Benoit TELLIER <bt...@apache.org>]

*# Context*

James does mostly 2 kinds of processing:

  - Mail processing: when receiving a mail in SMTP, the mail is enqueued 
and then the mailet processing is executed. Mailet/matchers are called 
against it and a serie of decision can be made: store this mail n a user 
mailbox, forward it, bounce, ignore it, etc...
  - Event processing: Once actions are taken in a user mailbox, an event 
is emitted to the event bus. Listeners are then called to "decorate" 
features of the mailbox manager in a non invasive way. ElasticSearch 
indexing, quota management, JMAP projections and state, IMAP/JMAP 
notifications...

Of course each of these processing can fail and error management is applied.

Regarding Mail processing the mail is stored in /var/mail/error .
To detect incidents:
   - ERROR logs during processing
   - Webadmin calls shows a non-zero /var/mail/error repository size
To fix this incident:
   - Explicit admin action is required, and if needed a reprocessing can 
be attempted (webadmin)

Regarding Event processing, listener execution is retried several time 
with a delay. If it keeps failingit is eventually stored in dead letter.
To detect incidents:
  - ERROR logs during processing
  - WebAdmin reports a non-zero size for deadletter
  - Health check, wich eventually does a recuring WARNING log that 
cannot be missed.
To fix this incident:
   - Explicit admin action is required, and if needed a redelivery can 
be attempted (webadmin)
*
**# Problem statement*

Most users misses this yet critical part of error management in James.

Actions are never taken, problems piles up.

While understandably major incidents with thousands of problems would 
clearly benefit from an admin intervention, I would like small incidents 
to self recover without a human intervention.

In practice, none of my clients (me included) managed to set up a 
reliable action plan regarding processing failures. Problems could be 
takled months after they arise thus escalating in major issues needlessly.

*# Proposal*

Get a recuring job addressing reprocessing/redelivery.

In order to avoid a retry storm, propose an upper bound to the count of 
entities reprocessed/redelivered. Such a retry storm could occur upon 
large scale incidents or persisting failures.

*# Impact*

In practice one reprocessing/redelivery per hour of 10 elements maximum 
would enable auto-recovering from minor incidents, without endangering 
overall performances, and would greatly benefit overall stability.

We thus would achieve auto-recovery of minor incidents.

Major incidents / persisting failures would still require admin actions.

*# Alternative*

An upper bound on redelivery/reprocessing exposed by webadmin would 
enpower the same kind of behaviour with external CRON.

Benefits:
  - Simpler code in James code base
  - Too easy for an admin not to configure this CRON

If the main proposal is deemed too complex, then this one would still 
fit my needs. I would just be happy to be selling consulting to say a 
user to configure these CRON ;-)

*# Related topics**
*
1.We would benefit from having the following metrics:

  - Size of deadletter over time
  - Size of mail repository over time

It would enable the set up of Grafana boards, allowing to detect issues 
without grocking logs.

2. We could have a health check checking /var/mail/error size and 
submitting a recuring WARNING.

In practice this proved useful to diagnose deadletter issues. I think 
error mail repository would benefit from this too.
*
**# Follow up*

I would be happy to open an ADR following this mailing thread. I can 
open JIRAs for the selected solutions,.

As we plan to do a Polish Sprint in June, Linagora would have some 
bandwith to work on the topic then.

----------------

Best regards,

Benoit