You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by bt...@apache.org on 2020/05/29 03:21:50 UTC
[james-project] branch master updated (8188037 -> cccbdd7)
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git.
from 8188037 JAMES-3093 Add BasicAuthenticationStrategy for JMAP RFC-8621
new ca08ef2 [ADR] Handle LDAP product variation via module choice
new cccbdd7 [ADR] Write a standalone EventBus ADR to centralize and detail related knowledge
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...oving-configured-additional-mailboxListeners.md | 3 ++
.../0027-eventBus-error-handling-upon-dispatch.md | 2 +
.../0035-distributed-listeners-configuration.md | 4 +-
...e-of-conditional-statements-in-guice-modules.md | 9 +++-
src/adr/0036-eventbus.md | 63 ++++++++++++++++++++++
src/adr/0037-distributed-eventbus.md | 46 ++++++++++++++++
6 files changed, 125 insertions(+), 2 deletions(-)
create mode 100644 src/adr/0036-eventbus.md
create mode 100644 src/adr/0037-distributed-eventbus.md
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 01/02: [ADR] Handle LDAP product variation via
module choice
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit ca08ef2c3c052afac95777216ba3ce5b19d253b0
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Thu May 14 12:21:24 2020 +0700
[ADR] Handle LDAP product variation via module choice
---
...036-against-use-of-conditional-statements-in-guice-modules.md | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/src/adr/0036-against-use-of-conditional-statements-in-guice-modules.md b/src/adr/0036-against-use-of-conditional-statements-in-guice-modules.md
index efc0bdb..55307e7 100644
--- a/src/adr/0036-against-use-of-conditional-statements-in-guice-modules.md
+++ b/src/adr/0036-against-use-of-conditional-statements-in-guice-modules.md
@@ -82,6 +82,9 @@ The following modules perform conditional statements upon injection time:
- BlobStoreChoosingModule::provideBlobStore: Choice of BlobStore technology: Cassandra, ObjectStorage or Hybrid
- [Cached blob store](https://github.com/linagora/james-project/pull/3319) represents a similar problem: should the
blobStore be wrapped by a caching layer?
+
+Cassandra and Distributed products are furthermore duplicated to offer a version supporting LDAP authentication. JPA
+product does not offer LDAP support.
## Decision
@@ -111,4 +114,8 @@ The following conditional statements in guice modules needs to be removed :
ObjectStorageDependenciesModule::selectBlobStoreBuilder and Cassandra Blob Store Cache conditional statement.
- [S3 native blobStore implementation](https://github.com/linagora/james-project/pull/3099) along side with S3 endpoints
support as part of Swift removes the need to select the Object Storage implementation.
- - Follow up work needs to be plan concerning `BlobExportMechanismModule` and `TikaMailboxModule::provideTextExtractor`.
\ No newline at end of file
+ - Follow up work needs to be plan concerning `BlobExportMechanismModule` and `TikaMailboxModule::provideTextExtractor`.
+
+We furthermore need to enable a module choice for LDAP on top of other existing products. We should remove LDAP variations
+for LDAP products. Corresponding docker image will be based on their non LDAP version, overriding the `usersrepository.xml`
+configuration file, be marked as deprecated and eventually removed.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
[james-project] 02/02: [ADR] Write a standalone EventBus ADR to
centralize and detail related knowledge
Posted by bt...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
commit cccbdd788e29c9dc796a56591ec40f21ed021788
Author: Benoit Tellier <bt...@linagora.com>
AuthorDate: Tue May 5 16:45:38 2020 +0700
[ADR] Write a standalone EventBus ADR to centralize and detail related knowledge
---
...oving-configured-additional-mailboxListeners.md | 3 ++
.../0027-eventBus-error-handling-upon-dispatch.md | 2 +
.../0035-distributed-listeners-configuration.md | 4 +-
src/adr/0036-eventbus.md | 63 ++++++++++++++++++++++
src/adr/0037-distributed-eventbus.md | 46 ++++++++++++++++
5 files changed, 117 insertions(+), 1 deletion(-)
diff --git a/src/adr/0026-removing-configured-additional-mailboxListeners.md b/src/adr/0026-removing-configured-additional-mailboxListeners.md
index d1c8ceb..f81c59b 100644
--- a/src/adr/0026-removing-configured-additional-mailboxListeners.md
+++ b/src/adr/0026-removing-configured-additional-mailboxListeners.md
@@ -18,6 +18,9 @@ per listener.
The distributed James server then declares a queue upon start for each one of these user registered listeners, that it
binds to the main event exchange.
+More information about this component, and its distributed, RabbitMQ based implementation, can be found in
+[ADR 0036](0036-eventbus.md).
+
If the user unconfigures the listener, the queue and the binding are still present but not consumed. This results in
unbounded queue growth eventually causing RabbitMQ resource exhaustion and failure.
diff --git a/src/adr/0027-eventBus-error-handling-upon-dispatch.md b/src/adr/0027-eventBus-error-handling-upon-dispatch.md
index 60e5cba..053b713 100644
--- a/src/adr/0027-eventBus-error-handling-upon-dispatch.md
+++ b/src/adr/0027-eventBus-error-handling-upon-dispatch.md
@@ -19,6 +19,8 @@ order to retry their processing via webAdmin APIs.
However no such capabilities are supported upon dispatching the event on the eventbus. A failed dispatch will result in message loss.
+More information about this component can be found in [ADR 0036](0036-eventbus.md).
+
## Decision
Upon dispatch failure, the eventBus should save events in dead letter using a dedicated group.
diff --git a/src/adr/0035-distributed-listeners-configuration.md b/src/adr/0035-distributed-listeners-configuration.md
index 626dfbb..78ada23 100644
--- a/src/adr/0035-distributed-listeners-configuration.md
+++ b/src/adr/0035-distributed-listeners-configuration.md
@@ -19,6 +19,8 @@ Mailbox listeners can be registered to be triggered every time an event is gener
mailbox. They are being executed in a distributed fashion following the workqueue messaging pattern. The "group" is an
attribute of the mailbox listener identifying to which work queue they belong.
+More information about this component can be found in [ADR 0036](0036-eventbus.md).
+
Currently, mailbox listeners are determined by the guice bindings of the server and additional mailbox listeners defined
via configuration files.
@@ -76,7 +78,7 @@ A WebAdmin endpoint will allow:
A broadcast on the event bus will be attempted to propagate topology changes, by the mean of a common registrationKey
to all nodes, a "TopologyChanged" event, and a mailbox listener starting the MailboxListeners on local node upon
-topology changes.
+topology changes. `registrationKey` concept is explained in [ADR 0036](0036-eventbus.md).
If a listener is added but is not in the classpath, an ERROR log is emitted. This can happen during a rolling upgrade,
which defines a new guice binding for a new mailbox listener. Events will still be emitted (and consumed by other James
diff --git a/src/adr/0036-eventbus.md b/src/adr/0036-eventbus.md
new file mode 100644
index 0000000..e16f782
--- /dev/null
+++ b/src/adr/0036-eventbus.md
@@ -0,0 +1,63 @@
+# 36. Event bus
+
+Date: 2020-05-05
+
+## Status
+
+Accepted (lazy consensus)
+
+## Context
+
+Many features rely on behaviors triggered by interactions with the mailbox API main interfaces (`RightManager`,
+`MailboxManager`, `MessageManager`, `MessageIdManager`). We need to provide a convenient extension mechanism for
+organizing the execution of these behaviours, provide retries and advanced error handling.
+
+Also, protocols enable notifications upon mailbox modifications. This is for instance the case for `RFC-2177 IMAP IDLE`,
+leveraged for `RFC-3501 IMAP unsolicitated notifications` when selecting a Mailbox, as well as maintaining the
+`IMAP Message Sequence Number <-> Unique IDentifier` MSN <-> UID mapping. Changes happening for a specific entity
+(mailbox) need to be propagated to the relevant listeners.
+
+## Decision
+
+James mailbox component, a core component of James handling the storage of mails and mailboxes, should use an event
+driven architecture.
+
+It means every meaningful action on mailboxes or messages triggers an event for any component to react to that event.
+
+`MailboxListener` allows executing actions upon mailbox events. They could be used for a wide variety of purposes, like
+enriching mailbox managers features or enabling user notifications upon mailboxes operations performed by other devices
+via other protocol sessions.
+
+Interactions happen via the managers (`RightManager`, `MailboxManager`, `MessageManager`, `MessageIdManager`) which emit an
+event on the `EventBus`, which will ensure the relevant `MailboxListener`s will be executed at least once.
+
+`MailboxListener` can be registered in a work queue fashion on the `EventBus`. Each work queue corresponds to a given
+MailboxListener class with the same configuration, identified by their group. Each event is executed at least once
+within a James cluster, errors are retried with an exponential back-off delay. If the execution keeps failing, the event
+ is stored in `DeadLetter` for later reprocessing, triggered via WebAdmin.
+
+Guice products enable the registration of additional mailbox listeners. A user can furthermore define its own
+mailboxListeners via the use of `extension-jars`.
+
+MailboxListener can also be registered to be executed only on events concerning a specific entity (eg. a mailbox). The
+`registrationKey` is identifying entities concerned by the event. Upon event emission, the manager will indicate the
+`registrationKey` this event should be sent to. A mailboxListener will thus only receive the event for the registration
+key it is registered to, in an at least once fashion.
+
+## Consequences
+
+We need to provide an `In VM` implementation of the EventBus for single server deployments.
+
+We also need to provide [a distributed event bus implementation](0037-distributed-eventbus.md).
+
+## Current usages
+
+The following features are implemented as Group mailbox listeners:
+
+ - Email indexing in Lucene or ElasticSearch
+ - Deletion of mailbox annotations
+ - Cassandra Message metadata cleanup upon deletion
+ - Quota updates
+ - Quota indexing
+ - Over Quota mailing
+ - SpamAssassin Spam/Ham reporting
diff --git a/src/adr/0037-distributed-eventbus.md b/src/adr/0037-distributed-eventbus.md
new file mode 100644
index 0000000..eea132e
--- /dev/null
+++ b/src/adr/0037-distributed-eventbus.md
@@ -0,0 +1,46 @@
+# 37. Distributed Event bus
+
+Date: 2020-05-25
+
+## Status
+
+Accepted (lazy consensus)
+
+## Context
+
+Read [Event Bus ADR](0036-eventbus.md) for context.
+
+Given several James servers, we need them to share a common EventBus.
+
+This:
+ - Ensures a better load balancing for `group mailbox listners`.
+ - Is required for correctness of notifications (like IMAP IDLE).
+
+## Decision
+
+Provide a distributed implementation of the EventBus leveraging RabbitMQ.
+
+Events are emitted to a single Exchange.
+
+Each group will have a corresponding queue, bound to the main exchange, with a default routing key. Each eventBus
+will consume this queue and execute the relevant listener, ensuring at least once execution at the cluster level.
+
+Retries are managed via a dedicated exchange for each group: as we need to count retries, the message headers need to
+be altered and we cannot rely on rabbitMQ build in retries. Each time the execution fails locally, a new event is emitted
+via the dedicated exchange, and the original event is acknowledged.
+
+Each eventBus will have a dedicated exclusive queue, bound to the main exchange with the `registrationKeys` used by local
+notification mailboxListeners (to only receive the corresponding subset of events). Errors are not retried for
+notifications, failures are not persisted within `DeadLetter`, achieving at most once event delivery.
+
+## Related ADRs
+
+The implementation of the the distributed EventBus suffers from the following flows:
+
+ - [Removing a configured additional MailboxListener](0026-removing-configured-additional-mailboxListeners.md)
+ - [Distributed Mailbox Listeners Configuration](0035-distributed-listeners-configuration.md) also covers more in details
+ topology changes and supersedes ADR 0026.
+
+The following enhancement have furthermore been contributed:
+
+ - [EventBus error handling upon dispatch](0027-eventBus-error-handling-upon-dispatch.md)
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org