Hexagonal Architecture - definitions of fundamental concepts

[David Leangen <ap...@leangen.net>]

Hi,

I created the issue: https://issues.apache.org/jira/browse/JAMES-3190

> I was told [1] that James has a hexogonal architecture [2]. However, that is not at all apparent neither in the documentation, nor in the code itself.
> If this is truly the fundamental concept of the system design, then for developers to understand the system more easily it ought to be made more explicit in the code, in one way or another (or in as many ways as reasonably possible).

[1] https://www.mail-archive.com/server-dev@james.apache.org/msg65809.html
[2] https://alistair.cockburn.us/hexagonal-architecture/


The discussion I wanted to have here is about what should be done about this. The email discussion was the only place that I have found so far where this is discussed. If it is such a fundamental concept, it really ought to be better captured.

First, I think it would be good to get agreement about what is:

 * Core
 * Port
 * Adapter
 * External Device

If we can reach agreement on a definition for each of these, then we can determine (1) how to document the concepts, and (2) if the code should be refactored to better reflect these concepts.

I will give it a go initially based on [2].

Core
-------------------------------
The core represents the business logic. It is what in Domain-Driven Design is called the “core domain”.

 * Examples: 
    - a Mailbox
    - a Mailbox admin (to create, rename, remove… a Mailbox)
    - a Mailbox Repository (to use the DDD concept, for providing persistence / storage)


Port
-------------------------------
The interface (in java) between the core and anything outside the core.

 * Examples: 
     - the java API of the Mailbox admin
     - the java API of the Mailbox Repository

Note that all External Devices would use the SAME ports. There is no specific port for a given technology.

Adapter
-------------------------------
A technology-specific implementation that has for its sole purpose the translation of a port into a specific technology.

  * Examples: 
     - a CLI for providing command-line access to the Mailbox admin API
     - a REST interface for providing HTTP access to the Mailbox admin API
     - a JDBC implementation of the Mailbox Repository


External Device
-------------------------------
Anything outside of the core that interacts with the core via a port.

  * Examples:
     - A GUI for providing a means of administering Mailboxes
     - A Derby implementation that implements the Mailbox Repository via the JDBC Adapter


Very short and sweet. Any comments?


Cheers,
=David



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James architecture: core components, sub-components and utility components

[Eugen Stan <eu...@netdava.com>]

Hi Benoît,

Thanks for the effort you put into this. It's very useful.

I also saw the list of documents
https://github.com/apache/james-project/tree/master/src/adr .  I need to
go through them.

@Raphaël : IMO there are many points of view here:

- the technical / developer view - we work on source to produce
libraries and apps

- the business / product view - we package the code in binary form to
distribute it to our users.

The same code can be packaged in many ways and in many packages (think
products in a gift basket :) ) .

We get to decide what we deliver out of the box and in what packages.

We don't have to build a single package with everything.

It's more valuable to have them easily composable.

Having good documentation is paramount here. Since with docs, we empower
users to take care of themselves, instead of relying on us to provide
them with everything.

We can maintain a few common James offerings - for common use cases /
configurations. We should encourage users to customize their build and
we should point them at the docs.


Again, awesome job Benoît .


La 11.06.2020 18:54, Raphaël Ouazana-Sustowski a scris:
> Hello,
>
> Even if I globally agree with your definition of extension, I think
> import/export is a very valuable feature and should be promoted (when
> finished). Of course from an architecture point of view it could be
> seen as an extension but for an admin it should be well integrated and
> easy to use.
>
> Regards,
>
> Raphaël.
>
> Le 11/06/2020 à 08:21, Tellier Benoit a écrit :
>>
>> On 10/06/2020 22:49, Raphaël Ouazana-Sustowski wrote:
>>> Hi Benoît,
>>>
>>> This inventory is amazing, thanks for having issued it.
>>>
>>> For my part it's just missing the LDAP read only user repository as an
>>> implementation of the user repository.
>> True.
>>
>>> But I probably missed things :)
>>>
>>> Oh for example the (unfinished?) import / export mechanism.
>> I would count it as a mailbox extension. I did not listed them on
>> purpose as I think these subproject parts address very specific needs
>> and should not be packaged as part of out-of-the-box servers.
>>
>> I rather think that each server should have a list of extensions that we
>> can use with it.
>>
>> So far I can come up with the following list of extensions candidates
>> (not only mailbox ones):
>>   - Data Leak Prevention
>>   - SpamAssassin
>>   - OverQuota mailing
>>   - Quota search
>>   - Of course mailbox import/export
>>   - DeletedMessages vault
>>   - Maybe even Sieve support could live as an extension?
>>   - WKD would IMO also qualify...
>>   - ...
>>
>> As a regular user, I don't really care about them. They should not
>> complexify the configuration, documentation or the server itself that I
>> am running.
>>
>> I started detailing my thoughts about extension mechanism in this email
>> https://www.mail-archive.com/server-dev@james.apache.org/msg66206.html
>>
>> I believe that extracting extensions out of default servers would ease
>> reducing build time.
>>
>> But I also believe it would help us building (guice based) servers that
>> are easier to extend.
>>
>> Regards,
>>
>> Benoit
>>
>>> Regards,
>>>
>>> Raphaël.
>>>
>>> Le 25/05/2020 à 07:00, Tellier Benoit a écrit :
>>>> Hi David,
>>>>
>>>> I gave a shot at defining:
>>>>
>>>>    - Core components, that offers services at heart of James
>>>>    - Utility components, that offers services core-components
>>>> implementations can rely on.
>>>>    - Mailbox sub-components.
>>>>
>>>> Here is the result. I encourage other project members to review it.
>>>>
>>>>    - Did I forgot some components?
>>>>    - Is the way I split them relevant?
>>>>    - Is the way to describe a component (descriptions, entities, key
>>>> interfaces, plugin) relevant? Would you add/remove something?
>>>>    - Of course all the component level details...
>>>>
>>>> I hope it helps your documentation effort.
>>>>
>>>> Best regards,
>>>>
>>>> Benoit
>>>>
>>>> -------------------------------------
>>>>
>>>> # Core components
>>>>
>>>> ## MailetContainer
>>>>
>>>> Enables mail processing.
>>>>
>>>> Entity:
>>>>    - **Mails**: Message transiting between a sender and recipients
>>>>
>>>> Key interfaces:
>>>>    - **Mailet**: Enables to act upon an email: modify it, trigger side
>>>> effects
>>>>    - **Matcher**: Condition to trigger a mailet action
>>>>
>>>> Plugin: A user can register their own mailets / matchers
>>>>
>>>> Here is a list of mailets providing central services, and bridges
>>>> other
>>>> core components defined here after :
>>>>
>>>>    - **RemoteDelivery** enables sending emails to other mail servers,
>>>> using the SMTP protocol.
>>>>    - **LocalDelivery** enables delivering emails to local user
>>>> mailboxes.
>>>>    - **RecipientRewriteTable** queries and applied recipient
>>>> rewritting
>>>> rules.
>>>>    - **ToMailRepository** stores emails in a given mail repository.
>>>>    - **Sieve** executes stored sieve scripts against incoming emails.
>>>>
>>>> ## MailQueue
>>>>
>>>> Enables asynchronous mail processing. Enables review and
>>>> administration
>>>> of mail traffic awaiting processing.
>>>>
>>>> Entity: **Mails**
>>>>
>>>> Key interface: **MailQueue**, **ManageableMailQueue**
>>>>
>>>> Implementations:
>>>>    - Memory
>>>>    - (embedded) ActiveMQ
>>>>    - Distributed (RabbitMQ + Cassandra)
>>>>    - Files
>>>>
>>>> ## MailRepositories
>>>>
>>>> Enables storing emails, along with their processing context. Enables
>>>> traffic review.
>>>>
>>>> Can be used to recover error, stores spam, etc...
>>>>
>>>> Entity: **Mails**
>>>>
>>>> Key interfaces:
>>>>    - **MailRepository**
>>>>    - **MailRepositoryStore**: enables instanciation of
>>>> MailRepositories
>>>>
>>>> Plugins:
>>>>    - A user can register new classes of MailRepository, that he can
>>>> then
>>>> configure within it's Mailet Container
>>>>
>>>> Bundled implementation:
>>>>    - Memory
>>>>    - Cassandra
>>>>    - JDBC
>>>>    - File
>>>>
>>>> ## Mailbox
>>>>
>>>> The mailbox stores user messages for later retrieval.
>>>>
>>>> Entities:
>>>>    - **Messages**: a mime message belonging to a user, along with
>>>> their
>>>> mailbox context.
>>>>    - **Mailboxes**: a group of messages
>>>>
>>>> Sample operations:
>>>>    - Mailboxes can be created, deleted, renamed
>>>>    - Messages can be appended, deleted, moved, copied, their flags
>>>> can be
>>>> modified
>>>>
>>>> Key interfaces:
>>>>    - **MailboxManager**: Enable managing and accessing mailboxes
>>>>    - **MessageManager**: Enables accessing messages within a given
>>>> mailbox. Can be obtained via the MailboxManager.
>>>>    - **MessageIdManager**: Enables accessing messages by their unique
>>>> identifier.. Can be obtained via the MailboxManager.
>>>>
>>>> Implementations:
>>>>    - Memory implementation
>>>>    - Maildir implementation
>>>>    - JPA implementation
>>>>    - Cassandra implementation
>>>>
>>>> The mailbox components defines the following sub-components:
>>>>
>>>> ### The EventBus
>>>>
>>>> James mailbox uses an event driven architecture.
>>>>      It means every meaningful action on mailboxes or messages
>>>> triggers an
>>>> event for any component to react to that event.
>>>> `MailboxListener` allows executing actions upon mailbox events. They
>>>> could be used for a wide variety of purposes, like
>>>> enriching mailbox managers features or enabling user notifications
>>>> upon
>>>> concurrent mailboxes operations.
>>>>
>>>> Entities:
>>>>    - **Events**: describes an action that happenened on a mailbox
>>>>    - **MailboxListeners**: standard API for acting upon events.
>>>>
>>>> Key interfaces:
>>>>    - **EventBus**: enables MailboxListener registration and enables
>>>> dispatching events to the MailboxListeners
>>>>
>>>> Implementations:
>>>>    - In VM
>>>>    - RabbitMQ
>>>>
>>>> Plugins:
>>>>    - A user can register its own mailbox listeners.
>>>>
>>>> ### EventDeadLetter
>>>>
>>>> Failed event processing is being saved to the EventDeadLetter for both
>>>> diagnostic and reprocessing purposes.
>>>>
>>>> Entities:
>>>>    - **Events**: As per the EventBus
>>>>
>>>> Key interfaces:
>>>>    - **EventDeadLetter**
>>>>
>>>> Implementations:
>>>>    - Cassandra
>>>>    - Memory
>>>>
>>>> ### Search index
>>>>
>>>> Sub components allowing searching emails.
>>>>
>>>> Entities: **Messages**
>>>>
>>>> Key interfaces:
>>>>    - **MessageSearchIndex**
>>>>
>>>> Implementations:
>>>>    - Scrolling
>>>>    - Lucene
>>>>    - ElasticSearch
>>>>
>>>> ### Text extraction
>>>>
>>>> Allows extracting text from arbtrary files. It empowers a more
>>>> relevant
>>>> search within the MessageSearchIndex
>>>>
>>>> Key interfaces:
>>>>    - **TextExtractor**
>>>>
>>>> Implementations:
>>>>    - Default: only extracts plain text
>>>>    - JSoup: only extracts plain text or HTML
>>>>    - Tika: relias on Apache Tika
>>>>
>>>> ### Quotas
>>>>
>>>> Defines and enforces limitations on the mailbox resource usage. Keep
>>>> track of current resource usage.
>>>>
>>>> Entities:
>>>>    - Quota Root: defines a group of mailboxes for which a given quota
>>>> applies. James implementation defines QuotaRoot as the mailboxes
>>>> belonging to a user.
>>>>    - Resources: What resourc ehte quota tracks. Could be the count of
>>>> messages or the total sizes of messages.
>>>>    - Quota: A limit along with current usage for a given Quota Root.
>>>>
>>>> Key interfaces:
>>>>    - **QuotaManager** allows to define limits and retrieve quota
>>>> usage.
>>>>
>>>> ## data/api
>>>>
>>>> Stores server level metadata.
>>>>
>>>> Entities:
>>>>    - **Users**: people enables to access the server, along with their
>>>> credentials
>>>>    - **Domains**: Logical group of users
>>>>
>>>> Key interfaces:
>>>>    - **UsersRepository**
>>>>    - **DomainList**
>>>>
>>>> Implementation:
>>>>    - Memory
>>>>    - Cassandra
>>>>    - JPA
>>>>
>>>> ## Recipient Rewrite Tables
>>>>
>>>> Enables storing rules for rewriting recipient of an email.
>>>>
>>>> Entities:
>>>>    - **Mapping source**: defines which address should be rewritten
>>>> for a
>>>> given rule
>>>>    - **Mapping**: defines the rewriting that should be performed
>>>>
>>>> Key interface: **RecipientRewriteTable** stores all the Mappings along
>>>> with their sources.
>>>>
>>>> Implementations:
>>>>    - Memory
>>>>    - Cassandra
>>>>    - JPA
>>>>    - XML (configuration only)
>>>>
>>>> ## SieveRepository
>>>>
>>>> Enables users to store Sieve scripts. Enables managing quota
>>>> applied for
>>>> user Sieve scripts.
>>>>
>>>> Entity:
>>>>    - **Sieve script**: enable a user to define actions to be performed
>>>> upon mail reception.
>>>>
>>>> Key interfaces:
>>>>    - **SieveRepository**: Leverages sieve script storage
>>>>    - **SieveQuotaRepository**: Leverages sieve script quota storage
>>>>
>>>> ## DNS Service
>>>>
>>>> Provides abstraction for DNS resolutions. Helps DNS resolution for
>>>> mail
>>>> processing purposes.
>>>>
>>>> Key interfaces: **DNSService** enables DNS resolution.
>>>>
>>>> # Protocols
>>>>
>>>> ## SMTP
>>>>
>>>> Implementation of RFC-5321 to enable receiving emails. Mails are
>>>> processed asynchronously.
>>>>
>>>> A user can register additional **ProtocolHandlers** to extend the
>>>> capabilities of the SMTP server.
>>>>
>>>> A SMTP **Hook** is a specific Protocol handler being plugged in a
>>>> specific location within the default SMTP server stack.
>>>>
>>>> ## LMTP
>>>>
>>>> Implementation of RFC-2033 to enable receiving emails. Similar to SMTP
>>>> but mails are processed synchronously.
>>>>
>>>> A user can register additional **ProtocolHandlers** to extend the
>>>> capabilities of the LMTP server.
>>>>
>>>> A SMTP **Hook** is a specific Protocol handler being plugged in a
>>>> specific location within the default SMTP server stack.
>>>>
>>>> ## IMAP
>>>>
>>>> Implementationof RFC-3501 to enable a user to access and manage their
>>>> mailbox.
>>>>
>>>> A user as of today can not extend the behaviour of the IMAP stack.
>>>>
>>>> ## JMAP
>>>>
>>>> Implementation of RFC-8620 and RFC-8621 to enable a user to
>>>> interact and
>>>> manage their mailboxes, and to send emails, on top of HTTP.
>>>>
>>>> The implementation of these specifications is currently a work in
>>>> progress. A previous draft implementation is available.
>>>>
>>>> ## POP3
>>>>
>>>> Implementation of RFC-1939 to enables users to retrieve the mails
>>>> within
>>>> their mailboxes.
>>>>
>>>> A user can register additional **ProtocolHandlers** to extend the
>>>> capabilities of the POP3 server.
>>>>
>>>> ## ManagedSieve
>>>>
>>>> Implementation of RFC-5804 to enable a user to upload and manage their
>>>> Sieve scripts.
>>>>
>>>> This protocol implementation is known to be buggy.
>>>>
>>>> # Server Administration
>>>>
>>>> ## REST administration via WebAdmin
>>>>
>>>> James specific REST APIs to manage other components, and their
>>>> specific
>>>> implementations.
>>>>
>>>> WebAdmin routes are defined in a modular way, the webadmin API will
>>>> thus
>>>> be product specific.
>>>>
>>>> Specification:
>>>>
>>>> Key interface:
>>>>    - **Routes** enables registering additional REST endpoints.
>>>>
>>>> Plugin: a user can define custom webadmin routes.
>>>>
>>>> ## Administration via the CLI
>>>>
>>>> Command Line interface to manage James.
>>>>
>>>> This CLI is not product specific, some command will not work for some
>>>> products.
>>>>
>>>> The CLI relies on JMX protocol and can represent a potential
>>>> vulnerabilities. Guice products enables disabing it. A long term
>>>> work is
>>>> to port the CLI to rely on the WebAdmin REST APIs.
>>>>
>>>> # Utility components
>>>>
>>>> These components offers services core-components implementations can
>>>> rely on.
>>>>
>>>> ## BlobStore
>>>>
>>>> Stores potentially large binary data.
>>>>
>>>> Entity:
>>>>    - **Blob**: Binary data
>>>>
>>>> Key interface: BlobStore
>>>>
>>>> Implementations:
>>>>    - Memory
>>>>    - ObjectStorage (S3/Swift)
>>>>
>>>> ## TaskManager
>>>>
>>>> Allows to control and schedule long running tasks run by other
>>>> components. Among other it enables scheduling, progress monitoring,
>>>> cancelation of long running tasks
>>>>
>>>> Entity:
>>>>    - **Task**: An operation performed by the TaskManager
>>>>    - **Task additional information**: Task specific information.
>>>> Exposes
>>>> specific details about this task and how its execution went on.
>>>>
>>>> Key interface: **TaskManager**.
>>>>
>>>> Implementations:
>>>>    - Memory
>>>>    - Distributed (cassandra + RabbitMQ)
>>>>
>>>> ## Metrics
>>>>
>>>> Enables recording execution timing for various operation within James.
>>>> Enables fine grained performance monitoring of a running James server,
>>>> for example using grafana boards.
>>>>
>>>> Key interface: **MetricFactory**.
>>>>
>>>> Implementation:
>>>>    - Default
>>>>    - DropWizard
>>>>
>>>> ## HealthChecks
>>>>
>>>> Enables knowing the status of each components within a running James
>>>> server.
>>>>
>>>> This is both periodically logged, and exposed via WebAdmin.
>>>>
>>>> Key interface: **HealthCheck**.
>>>>
>>>> ## Event sourcing
>>>>
>>>> Event sourcing implementation for the James server. Enables components
>>>> to rely on event sourcing technics for taking decisions.
>>>>
>>>> Here the definition of event is not the one of the mailbox, we are
>>>> in a
>>>> different bounded context.
>>>>
>>>> Entity:
>>>>
>>>>    - **Aggregate** represents an entity on which actions are performed
>>>>    - **command** represents actions we wish to perform on this
>>>> aggregate
>>>>    - **events** represent modifications of the aggregate. The state
>>>> of the
>>>> aggregate can be computed from its event history.
>>>>    - **CommandHandler** applies commands on aggregates and
>>>> generates the
>>>> resulting events.
>>>>    - **Subscriber** enables applying actions upon events.
>>>>
>>>> Key interfaces:
>>>>
>>>>    - **EventStore** stores events generated by event sourcing.
>>>> Implemented
>>>> in Memory and on top of Cassandra.
>>>>
>>>> ## JSON serialization
>>>>
>>>> Generic mechanism for modular JSON serialization of entities.
>>>>
>>>> Sample usages:
>>>>    - Task
>>>>    - Task additional details
>>>>    - Event sourcing events
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
-- 
Eugen Stan
+40720 898 747 / netdava.com

Re: James architecture: core components, sub-components and utility components

[Raphaël Ouazana-Sustowski <ro...@apache.org>]

Hello,

Even if I globally agree with your definition of extension, I think 
import/export is a very valuable feature and should be promoted (when 
finished). Of course from an architecture point of view it could be seen 
as an extension but for an admin it should be well integrated and easy 
to use.

Regards,

Raphaël.

Le 11/06/2020 à 08:21, Tellier Benoit a écrit :
>
> On 10/06/2020 22:49, Raphaël Ouazana-Sustowski wrote:
>> Hi Benoît,
>>
>> This inventory is amazing, thanks for having issued it.
>>
>> For my part it's just missing the LDAP read only user repository as an
>> implementation of the user repository.
> True.
>
>> But I probably missed things :)
>>
>> Oh for example the (unfinished?) import / export mechanism.
> I would count it as a mailbox extension. I did not listed them on
> purpose as I think these subproject parts address very specific needs
> and should not be packaged as part of out-of-the-box servers.
>
> I rather think that each server should have a list of extensions that we
> can use with it.
>
> So far I can come up with the following list of extensions candidates
> (not only mailbox ones):
>   - Data Leak Prevention
>   - SpamAssassin
>   - OverQuota mailing
>   - Quota search
>   - Of course mailbox import/export
>   - DeletedMessages vault
>   - Maybe even Sieve support could live as an extension?
>   - WKD would IMO also qualify...
>   - ...
>
> As a regular user, I don't really care about them. They should not
> complexify the configuration, documentation or the server itself that I
> am running.
>
> I started detailing my thoughts about extension mechanism in this email
> https://www.mail-archive.com/server-dev@james.apache.org/msg66206.html
>
> I believe that extracting extensions out of default servers would ease
> reducing build time.
>
> But I also believe it would help us building (guice based) servers that
> are easier to extend.
>
> Regards,
>
> Benoit
>
>> Regards,
>>
>> Raphaël.
>>
>> Le 25/05/2020 à 07:00, Tellier Benoit a écrit :
>>> Hi David,
>>>
>>> I gave a shot at defining:
>>>
>>>    - Core components, that offers services at heart of James
>>>    - Utility components, that offers services core-components
>>> implementations can rely on.
>>>    - Mailbox sub-components.
>>>
>>> Here is the result. I encourage other project members to review it.
>>>
>>>    - Did I forgot some components?
>>>    - Is the way I split them relevant?
>>>    - Is the way to describe a component (descriptions, entities, key
>>> interfaces, plugin) relevant? Would you add/remove something?
>>>    - Of course all the component level details...
>>>
>>> I hope it helps your documentation effort.
>>>
>>> Best regards,
>>>
>>> Benoit
>>>
>>> -------------------------------------
>>>
>>> # Core components
>>>
>>> ## MailetContainer
>>>
>>> Enables mail processing.
>>>
>>> Entity:
>>>    - **Mails**: Message transiting between a sender and recipients
>>>
>>> Key interfaces:
>>>    - **Mailet**: Enables to act upon an email: modify it, trigger side
>>> effects
>>>    - **Matcher**: Condition to trigger a mailet action
>>>
>>> Plugin: A user can register their own mailets / matchers
>>>
>>> Here is a list of mailets providing central services, and bridges other
>>> core components defined here after :
>>>
>>>    - **RemoteDelivery** enables sending emails to other mail servers,
>>> using the SMTP protocol.
>>>    - **LocalDelivery** enables delivering emails to local user mailboxes.
>>>    - **RecipientRewriteTable** queries and applied recipient rewritting
>>> rules.
>>>    - **ToMailRepository** stores emails in a given mail repository.
>>>    - **Sieve** executes stored sieve scripts against incoming emails.
>>>
>>> ## MailQueue
>>>
>>> Enables asynchronous mail processing. Enables review and administration
>>> of mail traffic awaiting processing.
>>>
>>> Entity: **Mails**
>>>
>>> Key interface: **MailQueue**, **ManageableMailQueue**
>>>
>>> Implementations:
>>>    - Memory
>>>    - (embedded) ActiveMQ
>>>    - Distributed (RabbitMQ + Cassandra)
>>>    - Files
>>>
>>> ## MailRepositories
>>>
>>> Enables storing emails, along with their processing context. Enables
>>> traffic review.
>>>
>>> Can be used to recover error, stores spam, etc...
>>>
>>> Entity: **Mails**
>>>
>>> Key interfaces:
>>>    - **MailRepository**
>>>    - **MailRepositoryStore**: enables instanciation of MailRepositories
>>>
>>> Plugins:
>>>    - A user can register new classes of MailRepository, that he can then
>>> configure within it's Mailet Container
>>>
>>> Bundled implementation:
>>>    - Memory
>>>    - Cassandra
>>>    - JDBC
>>>    - File
>>>
>>> ## Mailbox
>>>
>>> The mailbox stores user messages for later retrieval.
>>>
>>> Entities:
>>>    - **Messages**: a mime message belonging to a user, along with their
>>> mailbox context.
>>>    - **Mailboxes**: a group of messages
>>>
>>> Sample operations:
>>>    - Mailboxes can be created, deleted, renamed
>>>    - Messages can be appended, deleted, moved, copied, their flags can be
>>> modified
>>>
>>> Key interfaces:
>>>    - **MailboxManager**: Enable managing and accessing mailboxes
>>>    - **MessageManager**: Enables accessing messages within a given
>>> mailbox. Can be obtained via the MailboxManager.
>>>    - **MessageIdManager**: Enables accessing messages by their unique
>>> identifier.. Can be obtained via the MailboxManager.
>>>
>>> Implementations:
>>>    - Memory implementation
>>>    - Maildir implementation
>>>    - JPA implementation
>>>    - Cassandra implementation
>>>
>>> The mailbox components defines the following sub-components:
>>>
>>> ### The EventBus
>>>
>>> James mailbox uses an event driven architecture.
>>>      
>>> It means every meaningful action on mailboxes or messages triggers an
>>> event for any component to react to that event.
>>> `MailboxListener` allows executing actions upon mailbox events. They
>>> could be used for a wide variety of purposes, like
>>> enriching mailbox managers features or enabling user notifications upon
>>> concurrent mailboxes operations.
>>>
>>> Entities:
>>>    - **Events**: describes an action that happenened on a mailbox
>>>    - **MailboxListeners**: standard API for acting upon events.
>>>
>>> Key interfaces:
>>>    - **EventBus**: enables MailboxListener registration and enables
>>> dispatching events to the MailboxListeners
>>>
>>> Implementations:
>>>    - In VM
>>>    - RabbitMQ
>>>
>>> Plugins:
>>>    - A user can register its own mailbox listeners.
>>>
>>> ### EventDeadLetter
>>>
>>> Failed event processing is being saved to the EventDeadLetter for both
>>> diagnostic and reprocessing purposes.
>>>
>>> Entities:
>>>    - **Events**: As per the EventBus
>>>
>>> Key interfaces:
>>>    - **EventDeadLetter**
>>>
>>> Implementations:
>>>    - Cassandra
>>>    - Memory
>>>
>>> ### Search index
>>>
>>> Sub components allowing searching emails.
>>>
>>> Entities: **Messages**
>>>
>>> Key interfaces:
>>>    - **MessageSearchIndex**
>>>
>>> Implementations:
>>>    - Scrolling
>>>    - Lucene
>>>    - ElasticSearch
>>>
>>> ### Text extraction
>>>
>>> Allows extracting text from arbtrary files. It empowers a more relevant
>>> search within the MessageSearchIndex
>>>
>>> Key interfaces:
>>>    - **TextExtractor**
>>>
>>> Implementations:
>>>    - Default: only extracts plain text
>>>    - JSoup: only extracts plain text or HTML
>>>    - Tika: relias on Apache Tika
>>>
>>> ### Quotas
>>>
>>> Defines and enforces limitations on the mailbox resource usage. Keep
>>> track of current resource usage.
>>>
>>> Entities:
>>>    - Quota Root: defines a group of mailboxes for which a given quota
>>> applies. James implementation defines QuotaRoot as the mailboxes
>>> belonging to a user.
>>>    - Resources: What resourc ehte quota tracks. Could be the count of
>>> messages or the total sizes of messages.
>>>    - Quota: A limit along with current usage for a given Quota Root.
>>>
>>> Key interfaces:
>>>    - **QuotaManager** allows to define limits and retrieve quota usage.
>>>
>>> ## data/api
>>>
>>> Stores server level metadata.
>>>
>>> Entities:
>>>    - **Users**: people enables to access the server, along with their
>>> credentials
>>>    - **Domains**: Logical group of users
>>>
>>> Key interfaces:
>>>    - **UsersRepository**
>>>    - **DomainList**
>>>
>>> Implementation:
>>>    - Memory
>>>    - Cassandra
>>>    - JPA
>>>
>>> ## Recipient Rewrite Tables
>>>
>>> Enables storing rules for rewriting recipient of an email.
>>>
>>> Entities:
>>>    - **Mapping source**: defines which address should be rewritten for a
>>> given rule
>>>    - **Mapping**: defines the rewriting that should be performed
>>>
>>> Key interface: **RecipientRewriteTable** stores all the Mappings along
>>> with their sources.
>>>
>>> Implementations:
>>>    - Memory
>>>    - Cassandra
>>>    - JPA
>>>    - XML (configuration only)
>>>
>>> ## SieveRepository
>>>
>>> Enables users to store Sieve scripts. Enables managing quota applied for
>>> user Sieve scripts.
>>>
>>> Entity:
>>>    - **Sieve script**: enable a user to define actions to be performed
>>> upon mail reception.
>>>
>>> Key interfaces:
>>>    - **SieveRepository**: Leverages sieve script storage
>>>    - **SieveQuotaRepository**: Leverages sieve script quota storage
>>>
>>> ## DNS Service
>>>
>>> Provides abstraction for DNS resolutions. Helps DNS resolution for mail
>>> processing purposes.
>>>
>>> Key interfaces: **DNSService** enables DNS resolution.
>>>
>>> # Protocols
>>>
>>> ## SMTP
>>>
>>> Implementation of RFC-5321 to enable receiving emails. Mails are
>>> processed asynchronously.
>>>
>>> A user can register additional **ProtocolHandlers** to extend the
>>> capabilities of the SMTP server.
>>>
>>> A SMTP **Hook** is a specific Protocol handler being plugged in a
>>> specific location within the default SMTP server stack.
>>>
>>> ## LMTP
>>>
>>> Implementation of RFC-2033 to enable receiving emails. Similar to SMTP
>>> but mails are processed synchronously.
>>>
>>> A user can register additional **ProtocolHandlers** to extend the
>>> capabilities of the LMTP server.
>>>
>>> A SMTP **Hook** is a specific Protocol handler being plugged in a
>>> specific location within the default SMTP server stack.
>>>
>>> ## IMAP
>>>
>>> Implementationof RFC-3501 to enable a user to access and manage their
>>> mailbox.
>>>
>>> A user as of today can not extend the behaviour of the IMAP stack.
>>>
>>> ## JMAP
>>>
>>> Implementation of RFC-8620 and RFC-8621 to enable a user to interact and
>>> manage their mailboxes, and to send emails, on top of HTTP.
>>>
>>> The implementation of these specifications is currently a work in
>>> progress. A previous draft implementation is available.
>>>
>>> ## POP3
>>>
>>> Implementation of RFC-1939 to enables users to retrieve the mails within
>>> their mailboxes.
>>>
>>> A user can register additional **ProtocolHandlers** to extend the
>>> capabilities of the POP3 server.
>>>
>>> ## ManagedSieve
>>>
>>> Implementation of RFC-5804 to enable a user to upload and manage their
>>> Sieve scripts.
>>>
>>> This protocol implementation is known to be buggy.
>>>
>>> # Server Administration
>>>
>>> ## REST administration via WebAdmin
>>>
>>> James specific REST APIs to manage other components, and their specific
>>> implementations.
>>>
>>> WebAdmin routes are defined in a modular way, the webadmin API will thus
>>> be product specific.
>>>
>>> Specification:
>>>
>>> Key interface:
>>>    - **Routes** enables registering additional REST endpoints.
>>>
>>> Plugin: a user can define custom webadmin routes.
>>>
>>> ## Administration via the CLI
>>>
>>> Command Line interface to manage James.
>>>
>>> This CLI is not product specific, some command will not work for some
>>> products.
>>>
>>> The CLI relies on JMX protocol and can represent a potential
>>> vulnerabilities. Guice products enables disabing it. A long term work is
>>> to port the CLI to rely on the WebAdmin REST APIs.
>>>
>>> # Utility components
>>>
>>> These components offers services core-components implementations can
>>> rely on.
>>>
>>> ## BlobStore
>>>
>>> Stores potentially large binary data.
>>>
>>> Entity:
>>>    - **Blob**: Binary data
>>>
>>> Key interface: BlobStore
>>>
>>> Implementations:
>>>    - Memory
>>>    - ObjectStorage (S3/Swift)
>>>
>>> ## TaskManager
>>>
>>> Allows to control and schedule long running tasks run by other
>>> components. Among other it enables scheduling, progress monitoring,
>>> cancelation of long running tasks
>>>
>>> Entity:
>>>    - **Task**: An operation performed by the TaskManager
>>>    - **Task additional information**: Task specific information. Exposes
>>> specific details about this task and how its execution went on.
>>>
>>> Key interface: **TaskManager**.
>>>
>>> Implementations:
>>>    - Memory
>>>    - Distributed (cassandra + RabbitMQ)
>>>
>>> ## Metrics
>>>
>>> Enables recording execution timing for various operation within James.
>>> Enables fine grained performance monitoring of a running James server,
>>> for example using grafana boards.
>>>
>>> Key interface: **MetricFactory**.
>>>
>>> Implementation:
>>>    - Default
>>>    - DropWizard
>>>
>>> ## HealthChecks
>>>
>>> Enables knowing the status of each components within a running James
>>> server.
>>>
>>> This is both periodically logged, and exposed via WebAdmin.
>>>
>>> Key interface: **HealthCheck**.
>>>
>>> ## Event sourcing
>>>
>>> Event sourcing implementation for the James server. Enables components
>>> to rely on event sourcing technics for taking decisions.
>>>
>>> Here the definition of event is not the one of the mailbox, we are in a
>>> different bounded context.
>>>
>>> Entity:
>>>
>>>    - **Aggregate** represents an entity on which actions are performed
>>>    - **command** represents actions we wish to perform on this aggregate
>>>    - **events** represent modifications of the aggregate. The state of the
>>> aggregate can be computed from its event history.
>>>    - **CommandHandler** applies commands on aggregates and generates the
>>> resulting events.
>>>    - **Subscriber** enables applying actions upon events.
>>>
>>> Key interfaces:
>>>
>>>    - **EventStore** stores events generated by event sourcing. Implemented
>>> in Memory and on top of Cassandra.
>>>
>>> ## JSON serialization
>>>
>>> Generic mechanism for modular JSON serialization of entities.
>>>
>>> Sample usages:
>>>    - Task
>>>    - Task additional details
>>>    - Event sourcing events
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James architecture: core components, sub-components and utility components

[Tellier Benoit <bt...@apache.org>]

On 10/06/2020 22:49, Raphaël Ouazana-Sustowski wrote:
> Hi Benoît,
> 
> This inventory is amazing, thanks for having issued it.
> 
> For my part it's just missing the LDAP read only user repository as an
> implementation of the user repository.

True.

> But I probably missed things :)
> 
> Oh for example the (unfinished?) import / export mechanism.

I would count it as a mailbox extension. I did not listed them on
purpose as I think these subproject parts address very specific needs
and should not be packaged as part of out-of-the-box servers.

I rather think that each server should have a list of extensions that we
can use with it.

So far I can come up with the following list of extensions candidates
(not only mailbox ones):
 - Data Leak Prevention
 - SpamAssassin
 - OverQuota mailing
 - Quota search
 - Of course mailbox import/export
 - DeletedMessages vault
 - Maybe even Sieve support could live as an extension?
 - WKD would IMO also qualify...
 - ...

As a regular user, I don't really care about them. They should not
complexify the configuration, documentation or the server itself that I
am running.

I started detailing my thoughts about extension mechanism in this email
https://www.mail-archive.com/server-dev@james.apache.org/msg66206.html

I believe that extracting extensions out of default servers would ease
reducing build time.

But I also believe it would help us building (guice based) servers that
are easier to extend.

Regards,

Benoit

> 
> Regards,
> 
> Raphaël.
> 
> Le 25/05/2020 à 07:00, Tellier Benoit a écrit :
>> Hi David,
>>
>> I gave a shot at defining:
>>
>>   - Core components, that offers services at heart of James
>>   - Utility components, that offers services core-components
>> implementations can rely on.
>>   - Mailbox sub-components.
>>
>> Here is the result. I encourage other project members to review it.
>>
>>   - Did I forgot some components?
>>   - Is the way I split them relevant?
>>   - Is the way to describe a component (descriptions, entities, key
>> interfaces, plugin) relevant? Would you add/remove something?
>>   - Of course all the component level details...
>>
>> I hope it helps your documentation effort.
>>
>> Best regards,
>>
>> Benoit
>>
>> -------------------------------------
>>
>> # Core components
>>
>> ## MailetContainer
>>
>> Enables mail processing.
>>
>> Entity:
>>   - **Mails**: Message transiting between a sender and recipients
>>
>> Key interfaces:
>>   - **Mailet**: Enables to act upon an email: modify it, trigger side
>> effects
>>   - **Matcher**: Condition to trigger a mailet action
>>
>> Plugin: A user can register their own mailets / matchers
>>
>> Here is a list of mailets providing central services, and bridges other
>> core components defined here after :
>>
>>   - **RemoteDelivery** enables sending emails to other mail servers,
>> using the SMTP protocol.
>>   - **LocalDelivery** enables delivering emails to local user mailboxes.
>>   - **RecipientRewriteTable** queries and applied recipient rewritting
>> rules.
>>   - **ToMailRepository** stores emails in a given mail repository.
>>   - **Sieve** executes stored sieve scripts against incoming emails.
>>
>> ## MailQueue
>>
>> Enables asynchronous mail processing. Enables review and administration
>> of mail traffic awaiting processing.
>>
>> Entity: **Mails**
>>
>> Key interface: **MailQueue**, **ManageableMailQueue**
>>
>> Implementations:
>>   - Memory
>>   - (embedded) ActiveMQ
>>   - Distributed (RabbitMQ + Cassandra)
>>   - Files
>>
>> ## MailRepositories
>>
>> Enables storing emails, along with their processing context. Enables
>> traffic review.
>>
>> Can be used to recover error, stores spam, etc...
>>
>> Entity: **Mails**
>>
>> Key interfaces:
>>   - **MailRepository**
>>   - **MailRepositoryStore**: enables instanciation of MailRepositories
>>
>> Plugins:
>>   - A user can register new classes of MailRepository, that he can then
>> configure within it's Mailet Container
>>
>> Bundled implementation:
>>   - Memory
>>   - Cassandra
>>   - JDBC
>>   - File
>>
>> ## Mailbox
>>
>> The mailbox stores user messages for later retrieval.
>>
>> Entities:
>>   - **Messages**: a mime message belonging to a user, along with their
>> mailbox context.
>>   - **Mailboxes**: a group of messages
>>
>> Sample operations:
>>   - Mailboxes can be created, deleted, renamed
>>   - Messages can be appended, deleted, moved, copied, their flags can be
>> modified
>>
>> Key interfaces:
>>   - **MailboxManager**: Enable managing and accessing mailboxes
>>   - **MessageManager**: Enables accessing messages within a given
>> mailbox. Can be obtained via the MailboxManager.
>>   - **MessageIdManager**: Enables accessing messages by their unique
>> identifier.. Can be obtained via the MailboxManager.
>>
>> Implementations:
>>   - Memory implementation
>>   - Maildir implementation
>>   - JPA implementation
>>   - Cassandra implementation
>>
>> The mailbox components defines the following sub-components:
>>
>> ### The EventBus
>>
>> James mailbox uses an event driven architecture.
>>     
>> It means every meaningful action on mailboxes or messages triggers an
>> event for any component to react to that event.
>> `MailboxListener` allows executing actions upon mailbox events. They
>> could be used for a wide variety of purposes, like
>> enriching mailbox managers features or enabling user notifications upon
>> concurrent mailboxes operations.
>>
>> Entities:
>>   - **Events**: describes an action that happenened on a mailbox
>>   - **MailboxListeners**: standard API for acting upon events.
>>
>> Key interfaces:
>>   - **EventBus**: enables MailboxListener registration and enables
>> dispatching events to the MailboxListeners
>>
>> Implementations:
>>   - In VM
>>   - RabbitMQ
>>
>> Plugins:
>>   - A user can register its own mailbox listeners.
>>
>> ### EventDeadLetter
>>
>> Failed event processing is being saved to the EventDeadLetter for both
>> diagnostic and reprocessing purposes.
>>
>> Entities:
>>   - **Events**: As per the EventBus
>>
>> Key interfaces:
>>   - **EventDeadLetter**
>>
>> Implementations:
>>   - Cassandra
>>   - Memory
>>
>> ### Search index
>>
>> Sub components allowing searching emails.
>>
>> Entities: **Messages**
>>
>> Key interfaces:
>>   - **MessageSearchIndex**
>>
>> Implementations:
>>   - Scrolling
>>   - Lucene
>>   - ElasticSearch
>>
>> ### Text extraction
>>
>> Allows extracting text from arbtrary files. It empowers a more relevant
>> search within the MessageSearchIndex
>>
>> Key interfaces:
>>   - **TextExtractor**
>>
>> Implementations:
>>   - Default: only extracts plain text
>>   - JSoup: only extracts plain text or HTML
>>   - Tika: relias on Apache Tika
>>
>> ### Quotas
>>
>> Defines and enforces limitations on the mailbox resource usage. Keep
>> track of current resource usage.
>>
>> Entities:
>>   - Quota Root: defines a group of mailboxes for which a given quota
>> applies. James implementation defines QuotaRoot as the mailboxes
>> belonging to a user.
>>   - Resources: What resourc ehte quota tracks. Could be the count of
>> messages or the total sizes of messages.
>>   - Quota: A limit along with current usage for a given Quota Root.
>>
>> Key interfaces:
>>   - **QuotaManager** allows to define limits and retrieve quota usage.
>>
>> ## data/api
>>
>> Stores server level metadata.
>>
>> Entities:
>>   - **Users**: people enables to access the server, along with their
>> credentials
>>   - **Domains**: Logical group of users
>>
>> Key interfaces:
>>   - **UsersRepository**
>>   - **DomainList**
>>
>> Implementation:
>>   - Memory
>>   - Cassandra
>>   - JPA
>>
>> ## Recipient Rewrite Tables
>>
>> Enables storing rules for rewriting recipient of an email.
>>
>> Entities:
>>   - **Mapping source**: defines which address should be rewritten for a
>> given rule
>>   - **Mapping**: defines the rewriting that should be performed
>>
>> Key interface: **RecipientRewriteTable** stores all the Mappings along
>> with their sources.
>>
>> Implementations:
>>   - Memory
>>   - Cassandra
>>   - JPA
>>   - XML (configuration only)
>>
>> ## SieveRepository
>>
>> Enables users to store Sieve scripts. Enables managing quota applied for
>> user Sieve scripts.
>>
>> Entity:
>>   - **Sieve script**: enable a user to define actions to be performed
>> upon mail reception.
>>
>> Key interfaces:
>>   - **SieveRepository**: Leverages sieve script storage
>>   - **SieveQuotaRepository**: Leverages sieve script quota storage
>>
>> ## DNS Service
>>
>> Provides abstraction for DNS resolutions. Helps DNS resolution for mail
>> processing purposes.
>>
>> Key interfaces: **DNSService** enables DNS resolution.
>>
>> # Protocols
>>
>> ## SMTP
>>
>> Implementation of RFC-5321 to enable receiving emails. Mails are
>> processed asynchronously.
>>
>> A user can register additional **ProtocolHandlers** to extend the
>> capabilities of the SMTP server.
>>
>> A SMTP **Hook** is a specific Protocol handler being plugged in a
>> specific location within the default SMTP server stack.
>>
>> ## LMTP
>>
>> Implementation of RFC-2033 to enable receiving emails. Similar to SMTP
>> but mails are processed synchronously.
>>
>> A user can register additional **ProtocolHandlers** to extend the
>> capabilities of the LMTP server.
>>
>> A SMTP **Hook** is a specific Protocol handler being plugged in a
>> specific location within the default SMTP server stack.
>>
>> ## IMAP
>>
>> Implementationof RFC-3501 to enable a user to access and manage their
>> mailbox.
>>
>> A user as of today can not extend the behaviour of the IMAP stack.
>>
>> ## JMAP
>>
>> Implementation of RFC-8620 and RFC-8621 to enable a user to interact and
>> manage their mailboxes, and to send emails, on top of HTTP.
>>
>> The implementation of these specifications is currently a work in
>> progress. A previous draft implementation is available.
>>
>> ## POP3
>>
>> Implementation of RFC-1939 to enables users to retrieve the mails within
>> their mailboxes.
>>
>> A user can register additional **ProtocolHandlers** to extend the
>> capabilities of the POP3 server.
>>
>> ## ManagedSieve
>>
>> Implementation of RFC-5804 to enable a user to upload and manage their
>> Sieve scripts.
>>
>> This protocol implementation is known to be buggy.
>>
>> # Server Administration
>>
>> ## REST administration via WebAdmin
>>
>> James specific REST APIs to manage other components, and their specific
>> implementations.
>>
>> WebAdmin routes are defined in a modular way, the webadmin API will thus
>> be product specific.
>>
>> Specification:
>>
>> Key interface:
>>   - **Routes** enables registering additional REST endpoints.
>>
>> Plugin: a user can define custom webadmin routes.
>>
>> ## Administration via the CLI
>>
>> Command Line interface to manage James.
>>
>> This CLI is not product specific, some command will not work for some
>> products.
>>
>> The CLI relies on JMX protocol and can represent a potential
>> vulnerabilities. Guice products enables disabing it. A long term work is
>> to port the CLI to rely on the WebAdmin REST APIs.
>>
>> # Utility components
>>
>> These components offers services core-components implementations can
>> rely on.
>>
>> ## BlobStore
>>
>> Stores potentially large binary data.
>>
>> Entity:
>>   - **Blob**: Binary data
>>
>> Key interface: BlobStore
>>
>> Implementations:
>>   - Memory
>>   - ObjectStorage (S3/Swift)
>>
>> ## TaskManager
>>
>> Allows to control and schedule long running tasks run by other
>> components. Among other it enables scheduling, progress monitoring,
>> cancelation of long running tasks
>>
>> Entity:
>>   - **Task**: An operation performed by the TaskManager
>>   - **Task additional information**: Task specific information. Exposes
>> specific details about this task and how its execution went on.
>>
>> Key interface: **TaskManager**.
>>
>> Implementations:
>>   - Memory
>>   - Distributed (cassandra + RabbitMQ)
>>
>> ## Metrics
>>
>> Enables recording execution timing for various operation within James.
>> Enables fine grained performance monitoring of a running James server,
>> for example using grafana boards.
>>
>> Key interface: **MetricFactory**.
>>
>> Implementation:
>>   - Default
>>   - DropWizard
>>
>> ## HealthChecks
>>
>> Enables knowing the status of each components within a running James
>> server.
>>
>> This is both periodically logged, and exposed via WebAdmin.
>>
>> Key interface: **HealthCheck**.
>>
>> ## Event sourcing
>>
>> Event sourcing implementation for the James server. Enables components
>> to rely on event sourcing technics for taking decisions.
>>
>> Here the definition of event is not the one of the mailbox, we are in a
>> different bounded context.
>>
>> Entity:
>>
>>   - **Aggregate** represents an entity on which actions are performed
>>   - **command** represents actions we wish to perform on this aggregate
>>   - **events** represent modifications of the aggregate. The state of the
>> aggregate can be computed from its event history.
>>   - **CommandHandler** applies commands on aggregates and generates the
>> resulting events.
>>   - **Subscriber** enables applying actions upon events.
>>
>> Key interfaces:
>>
>>   - **EventStore** stores events generated by event sourcing. Implemented
>> in Memory and on top of Cassandra.
>>
>> ## JSON serialization
>>
>> Generic mechanism for modular JSON serialization of entities.
>>
>> Sample usages:
>>   - Task
>>   - Task additional details
>>   - Event sourcing events
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James architecture: core components, sub-components and utility components

[Raphaël Ouazana-Sustowski <ro...@apache.org>]

Hi Benoît,

This inventory is amazing, thanks for having issued it.

For my part it's just missing the LDAP read only user repository as an 
implementation of the user repository.

But I probably missed things :)

Oh for example the (unfinished?) import / export mechanism.

Regards,

Raphaël.

Le 25/05/2020 à 07:00, Tellier Benoit a écrit :
> Hi David,
>
> I gave a shot at defining:
>
>   - Core components, that offers services at heart of James
>   - Utility components, that offers services core-components
> implementations can rely on.
>   - Mailbox sub-components.
>
> Here is the result. I encourage other project members to review it.
>
>   - Did I forgot some components?
>   - Is the way I split them relevant?
>   - Is the way to describe a component (descriptions, entities, key
> interfaces, plugin) relevant? Would you add/remove something?
>   - Of course all the component level details...
>
> I hope it helps your documentation effort.
>
> Best regards,
>
> Benoit
>
> -------------------------------------
>
> # Core components
>
> ## MailetContainer
>
> Enables mail processing.
>
> Entity:
>   - **Mails**: Message transiting between a sender and recipients
>
> Key interfaces:
>   - **Mailet**: Enables to act upon an email: modify it, trigger side effects
>   - **Matcher**: Condition to trigger a mailet action
>
> Plugin: A user can register their own mailets / matchers
>
> Here is a list of mailets providing central services, and bridges other
> core components defined here after :
>
>   - **RemoteDelivery** enables sending emails to other mail servers,
> using the SMTP protocol.
>   - **LocalDelivery** enables delivering emails to local user mailboxes.
>   - **RecipientRewriteTable** queries and applied recipient rewritting rules.
>   - **ToMailRepository** stores emails in a given mail repository.
>   - **Sieve** executes stored sieve scripts against incoming emails.
>
> ## MailQueue
>
> Enables asynchronous mail processing. Enables review and administration
> of mail traffic awaiting processing.
>
> Entity: **Mails**
>
> Key interface: **MailQueue**, **ManageableMailQueue**
>
> Implementations:
>   - Memory
>   - (embedded) ActiveMQ
>   - Distributed (RabbitMQ + Cassandra)
>   - Files
>
> ## MailRepositories
>
> Enables storing emails, along with their processing context. Enables
> traffic review.
>
> Can be used to recover error, stores spam, etc...
>
> Entity: **Mails**
>
> Key interfaces:
>   - **MailRepository**
>   - **MailRepositoryStore**: enables instanciation of MailRepositories
>
> Plugins:
>   - A user can register new classes of MailRepository, that he can then
> configure within it's Mailet Container
>
> Bundled implementation:
>   - Memory
>   - Cassandra
>   - JDBC
>   - File
>
> ## Mailbox
>
> The mailbox stores user messages for later retrieval.
>
> Entities:
>   - **Messages**: a mime message belonging to a user, along with their
> mailbox context.
>   - **Mailboxes**: a group of messages
>
> Sample operations:
>   - Mailboxes can be created, deleted, renamed
>   - Messages can be appended, deleted, moved, copied, their flags can be
> modified
>
> Key interfaces:
>   - **MailboxManager**: Enable managing and accessing mailboxes
>   - **MessageManager**: Enables accessing messages within a given
> mailbox. Can be obtained via the MailboxManager.
>   - **MessageIdManager**: Enables accessing messages by their unique
> identifier.. Can be obtained via the MailboxManager.
>
> Implementations:
>   - Memory implementation
>   - Maildir implementation
>   - JPA implementation
>   - Cassandra implementation
>
> The mailbox components defines the following sub-components:
>
> ### The EventBus
>
> James mailbox uses an event driven architecture.
> 	
> It means every meaningful action on mailboxes or messages triggers an
> event for any component to react to that event.
> `MailboxListener` allows executing actions upon mailbox events. They
> could be used for a wide variety of purposes, like
> enriching mailbox managers features or enabling user notifications upon
> concurrent mailboxes operations.
>
> Entities:
>   - **Events**: describes an action that happenened on a mailbox
>   - **MailboxListeners**: standard API for acting upon events.
>
> Key interfaces:
>   - **EventBus**: enables MailboxListener registration and enables
> dispatching events to the MailboxListeners
>
> Implementations:
>   - In VM
>   - RabbitMQ
>
> Plugins:
>   - A user can register its own mailbox listeners.
>
> ### EventDeadLetter
>
> Failed event processing is being saved to the EventDeadLetter for both
> diagnostic and reprocessing purposes.
>
> Entities:
>   - **Events**: As per the EventBus
>
> Key interfaces:
>   - **EventDeadLetter**
>
> Implementations:
>   - Cassandra
>   - Memory
>
> ### Search index
>
> Sub components allowing searching emails.
>
> Entities: **Messages**
>
> Key interfaces:
>   - **MessageSearchIndex**
>
> Implementations:
>   - Scrolling
>   - Lucene
>   - ElasticSearch
>
> ### Text extraction
>
> Allows extracting text from arbtrary files. It empowers a more relevant
> search within the MessageSearchIndex
>
> Key interfaces:
>   - **TextExtractor**
>
> Implementations:
>   - Default: only extracts plain text
>   - JSoup: only extracts plain text or HTML
>   - Tika: relias on Apache Tika
>
> ### Quotas
>
> Defines and enforces limitations on the mailbox resource usage. Keep
> track of current resource usage.
>
> Entities:
>   - Quota Root: defines a group of mailboxes for which a given quota
> applies. James implementation defines QuotaRoot as the mailboxes
> belonging to a user.
>   - Resources: What resourc ehte quota tracks. Could be the count of
> messages or the total sizes of messages.
>   - Quota: A limit along with current usage for a given Quota Root.
>
> Key interfaces:
>   - **QuotaManager** allows to define limits and retrieve quota usage.
>
> ## data/api
>
> Stores server level metadata.
>
> Entities:
>   - **Users**: people enables to access the server, along with their
> credentials
>   - **Domains**: Logical group of users
>
> Key interfaces:
>   - **UsersRepository**
>   - **DomainList**
>
> Implementation:
>   - Memory
>   - Cassandra
>   - JPA
>
> ## Recipient Rewrite Tables
>
> Enables storing rules for rewriting recipient of an email.
>
> Entities:
>   - **Mapping source**: defines which address should be rewritten for a
> given rule
>   - **Mapping**: defines the rewriting that should be performed
>
> Key interface: **RecipientRewriteTable** stores all the Mappings along
> with their sources.
>
> Implementations:
>   - Memory
>   - Cassandra
>   - JPA
>   - XML (configuration only)
>
> ## SieveRepository
>
> Enables users to store Sieve scripts. Enables managing quota applied for
> user Sieve scripts.
>
> Entity:
>   - **Sieve script**: enable a user to define actions to be performed
> upon mail reception.
>
> Key interfaces:
>   - **SieveRepository**: Leverages sieve script storage
>   - **SieveQuotaRepository**: Leverages sieve script quota storage
>
> ## DNS Service
>
> Provides abstraction for DNS resolutions. Helps DNS resolution for mail
> processing purposes.
>
> Key interfaces: **DNSService** enables DNS resolution.
>
> # Protocols
>
> ## SMTP
>
> Implementation of RFC-5321 to enable receiving emails. Mails are
> processed asynchronously.
>
> A user can register additional **ProtocolHandlers** to extend the
> capabilities of the SMTP server.
>
> A SMTP **Hook** is a specific Protocol handler being plugged in a
> specific location within the default SMTP server stack.
>
> ## LMTP
>
> Implementation of RFC-2033 to enable receiving emails. Similar to SMTP
> but mails are processed synchronously.
>
> A user can register additional **ProtocolHandlers** to extend the
> capabilities of the LMTP server.
>
> A SMTP **Hook** is a specific Protocol handler being plugged in a
> specific location within the default SMTP server stack.
>
> ## IMAP
>
> Implementationof RFC-3501 to enable a user to access and manage their
> mailbox.
>
> A user as of today can not extend the behaviour of the IMAP stack.
>
> ## JMAP
>
> Implementation of RFC-8620 and RFC-8621 to enable a user to interact and
> manage their mailboxes, and to send emails, on top of HTTP.
>
> The implementation of these specifications is currently a work in
> progress. A previous draft implementation is available.
>
> ## POP3
>
> Implementation of RFC-1939 to enables users to retrieve the mails within
> their mailboxes.
>
> A user can register additional **ProtocolHandlers** to extend the
> capabilities of the POP3 server.
>
> ## ManagedSieve
>
> Implementation of RFC-5804 to enable a user to upload and manage their
> Sieve scripts.
>
> This protocol implementation is known to be buggy.
>
> # Server Administration
>
> ## REST administration via WebAdmin
>
> James specific REST APIs to manage other components, and their specific
> implementations.
>
> WebAdmin routes are defined in a modular way, the webadmin API will thus
> be product specific.
>
> Specification:
>
> Key interface:
>   - **Routes** enables registering additional REST endpoints.
>
> Plugin: a user can define custom webadmin routes.
>
> ## Administration via the CLI
>
> Command Line interface to manage James.
>
> This CLI is not product specific, some command will not work for some
> products.
>
> The CLI relies on JMX protocol and can represent a potential
> vulnerabilities. Guice products enables disabing it. A long term work is
> to port the CLI to rely on the WebAdmin REST APIs.
>
> # Utility components
>
> These components offers services core-components implementations can
> rely on.
>
> ## BlobStore
>
> Stores potentially large binary data.
>
> Entity:
>   - **Blob**: Binary data
>
> Key interface: BlobStore
>
> Implementations:
>   - Memory
>   - ObjectStorage (S3/Swift)
>
> ## TaskManager
>
> Allows to control and schedule long running tasks run by other
> components. Among other it enables scheduling, progress monitoring,
> cancelation of long running tasks
>
> Entity:
>   - **Task**: An operation performed by the TaskManager
>   - **Task additional information**: Task specific information. Exposes
> specific details about this task and how its execution went on.
>
> Key interface: **TaskManager**.
>
> Implementations:
>   - Memory
>   - Distributed (cassandra + RabbitMQ)
>
> ## Metrics
>
> Enables recording execution timing for various operation within James.
> Enables fine grained performance monitoring of a running James server,
> for example using grafana boards.
>
> Key interface: **MetricFactory**.
>
> Implementation:
>   - Default
>   - DropWizard
>
> ## HealthChecks
>
> Enables knowing the status of each components within a running James server.
>
> This is both periodically logged, and exposed via WebAdmin.
>
> Key interface: **HealthCheck**.
>
> ## Event sourcing
>
> Event sourcing implementation for the James server. Enables components
> to rely on event sourcing technics for taking decisions.
>
> Here the definition of event is not the one of the mailbox, we are in a
> different bounded context.
>
> Entity:
>
>   - **Aggregate** represents an entity on which actions are performed
>   - **command** represents actions we wish to perform on this aggregate
>   - **events** represent modifications of the aggregate. The state of the
> aggregate can be computed from its event history.
>   - **CommandHandler** applies commands on aggregates and generates the
> resulting events.
>   - **Subscriber** enables applying actions upon events.
>
> Key interfaces:
>
>   - **EventStore** stores events generated by event sourcing. Implemented
> in Memory and on top of Cassandra.
>
> ## JSON serialization
>
> Generic mechanism for modular JSON serialization of entities.
>
> Sample usages:
>   - Task
>   - Task additional details
>   - Event sourcing events
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James architecture: core components, sub-components and utility components

[David Leangen <ap...@leangen.net>]

Hi Benoit:

> I gave a shot at defining:
> 
> - Core components, that offers services at heart of James
> - Utility components, that offers services core-components
> implementations can rely on.
> - Mailbox sub-components.


Wow, there is a lot here! It will take me some time to digest, but this is very helpful.

I will use this as one of my main references. My immediate problem is grasping the whole picture at once, but I will continue that discussion here:

  https://www.mail-archive.com/server-dev@james.apache.org/msg65876.html


I first need to get the entire picture a little clearer in my head, then I will come back here for sure. In the meantime, as you were requesting, if others could also chime in and fill in any missing blanks, that would be great.

Thanks again!
=David



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James architecture: core components, sub-components and utility components

[Matthieu Baechler <ma...@apache.org>]

I finally read it and it's very good (I have some comments but will
keep them for when this become a documentation as they are really
minor).

Thank you Benoit, I think it will help building a overall picture of
what James is and how it works.

-- Matthieu Baechler

On Mon, 2020-05-25 at 12:00 +0700, Tellier Benoit wrote:
> Hi David,
> 
> I gave a shot at defining:
> 
>  - Core components, that offers services at heart of James
>  - Utility components, that offers services core-components
> implementations can rely on.
>  - Mailbox sub-components.
> 
> Here is the result. I encourage other project members to review it.
> 
>  - Did I forgot some components?
>  - Is the way I split them relevant?
>  - Is the way to describe a component (descriptions, entities, key
> interfaces, plugin) relevant? Would you add/remove something?
>  - Of course all the component level details...
> 
> I hope it helps your documentation effort.
> 
> Best regards,
> 
> Benoit
> 
> -------------------------------------
> 
> # Core components
> 
> ## MailetContainer
> 
> Enables mail processing.
> 
> Entity:
>  - **Mails**: Message transiting between a sender and recipients
> 
> Key interfaces:
>  - **Mailet**: Enables to act upon an email: modify it, trigger side
> effects
>  - **Matcher**: Condition to trigger a mailet action
> 
> Plugin: A user can register their own mailets / matchers
> 
> Here is a list of mailets providing central services, and bridges
> other
> core components defined here after :
> 
>  - **RemoteDelivery** enables sending emails to other mail servers,
> using the SMTP protocol.
>  - **LocalDelivery** enables delivering emails to local user
> mailboxes.
>  - **RecipientRewriteTable** queries and applied recipient rewritting
> rules.
>  - **ToMailRepository** stores emails in a given mail repository.
>  - **Sieve** executes stored sieve scripts against incoming emails.
> 
> ## MailQueue
> 
> Enables asynchronous mail processing. Enables review and
> administration
> of mail traffic awaiting processing.
> 
> Entity: **Mails**
> 
> Key interface: **MailQueue**, **ManageableMailQueue**
> 
> Implementations:
>  - Memory
>  - (embedded) ActiveMQ
>  - Distributed (RabbitMQ + Cassandra)
>  - Files
> 
> ## MailRepositories
> 
> Enables storing emails, along with their processing context. Enables
> traffic review.
> 
> Can be used to recover error, stores spam, etc...
> 
> Entity: **Mails**
> 
> Key interfaces:
>  - **MailRepository**
>  - **MailRepositoryStore**: enables instanciation of MailRepositories
> 
> Plugins:
>  - A user can register new classes of MailRepository, that he can
> then
> configure within it's Mailet Container
> 
> Bundled implementation:
>  - Memory
>  - Cassandra
>  - JDBC
>  - File
> 
> ## Mailbox
> 
> The mailbox stores user messages for later retrieval.
> 
> Entities:
>  - **Messages**: a mime message belonging to a user, along with their
> mailbox context.
>  - **Mailboxes**: a group of messages
> 
> Sample operations:
>  - Mailboxes can be created, deleted, renamed
>  - Messages can be appended, deleted, moved, copied, their flags can
> be
> modified
> 
> Key interfaces:
>  - **MailboxManager**: Enable managing and accessing mailboxes
>  - **MessageManager**: Enables accessing messages within a given
> mailbox. Can be obtained via the MailboxManager.
>  - **MessageIdManager**: Enables accessing messages by their unique
> identifier.. Can be obtained via the MailboxManager.
> 
> Implementations:
>  - Memory implementation
>  - Maildir implementation
>  - JPA implementation
>  - Cassandra implementation
> 
> The mailbox components defines the following sub-components:
> 
> ### The EventBus
> 
> James mailbox uses an event driven architecture.
> 	
> It means every meaningful action on mailboxes or messages triggers an
> event for any component to react to that event.
> `MailboxListener` allows executing actions upon mailbox events. They
> could be used for a wide variety of purposes, like
> enriching mailbox managers features or enabling user notifications
> upon
> concurrent mailboxes operations.
> 
> Entities:
>  - **Events**: describes an action that happenened on a mailbox
>  - **MailboxListeners**: standard API for acting upon events.
> 
> Key interfaces:
>  - **EventBus**: enables MailboxListener registration and enables
> dispatching events to the MailboxListeners
> 
> Implementations:
>  - In VM
>  - RabbitMQ
> 
> Plugins:
>  - A user can register its own mailbox listeners.
> 
> ### EventDeadLetter
> 
> Failed event processing is being saved to the EventDeadLetter for
> both
> diagnostic and reprocessing purposes.
> 
> Entities:
>  - **Events**: As per the EventBus
> 
> Key interfaces:
>  - **EventDeadLetter**
> 
> Implementations:
>  - Cassandra
>  - Memory
> 
> ### Search index
> 
> Sub components allowing searching emails.
> 
> Entities: **Messages**
> 
> Key interfaces:
>  - **MessageSearchIndex**
> 
> Implementations:
>  - Scrolling
>  - Lucene
>  - ElasticSearch
> 
> ### Text extraction
> 
> Allows extracting text from arbtrary files. It empowers a more
> relevant
> search within the MessageSearchIndex
> 
> Key interfaces:
>  - **TextExtractor**
> 
> Implementations:
>  - Default: only extracts plain text
>  - JSoup: only extracts plain text or HTML
>  - Tika: relias on Apache Tika
> 
> ### Quotas
> 
> Defines and enforces limitations on the mailbox resource usage. Keep
> track of current resource usage.
> 
> Entities:
>  - Quota Root: defines a group of mailboxes for which a given quota
> applies. James implementation defines QuotaRoot as the mailboxes
> belonging to a user.
>  - Resources: What resourc ehte quota tracks. Could be the count of
> messages or the total sizes of messages.
>  - Quota: A limit along with current usage for a given Quota Root.
> 
> Key interfaces:
>  - **QuotaManager** allows to define limits and retrieve quota usage.
> 
> ## data/api
> 
> Stores server level metadata.
> 
> Entities:
>  - **Users**: people enables to access the server, along with their
> credentials
>  - **Domains**: Logical group of users
> 
> Key interfaces:
>  - **UsersRepository**
>  - **DomainList**
> 
> Implementation:
>  - Memory
>  - Cassandra
>  - JPA
> 
> ## Recipient Rewrite Tables
> 
> Enables storing rules for rewriting recipient of an email.
> 
> Entities:
>  - **Mapping source**: defines which address should be rewritten for
> a
> given rule
>  - **Mapping**: defines the rewriting that should be performed
> 
> Key interface: **RecipientRewriteTable** stores all the Mappings
> along
> with their sources.
> 
> Implementations:
>  - Memory
>  - Cassandra
>  - JPA
>  - XML (configuration only)
> 
> ## SieveRepository
> 
> Enables users to store Sieve scripts. Enables managing quota applied
> for
> user Sieve scripts.
> 
> Entity:
>  - **Sieve script**: enable a user to define actions to be performed
> upon mail reception.
> 
> Key interfaces:
>  - **SieveRepository**: Leverages sieve script storage
>  - **SieveQuotaRepository**: Leverages sieve script quota storage
> 
> ## DNS Service
> 
> Provides abstraction for DNS resolutions. Helps DNS resolution for
> mail
> processing purposes.
> 
> Key interfaces: **DNSService** enables DNS resolution.
> 
> # Protocols
> 
> ## SMTP
> 
> Implementation of RFC-5321 to enable receiving emails. Mails are
> processed asynchronously.
> 
> A user can register additional **ProtocolHandlers** to extend the
> capabilities of the SMTP server.
> 
> A SMTP **Hook** is a specific Protocol handler being plugged in a
> specific location within the default SMTP server stack.
> 
> ## LMTP
> 
> Implementation of RFC-2033 to enable receiving emails. Similar to
> SMTP
> but mails are processed synchronously.
> 
> A user can register additional **ProtocolHandlers** to extend the
> capabilities of the LMTP server.
> 
> A SMTP **Hook** is a specific Protocol handler being plugged in a
> specific location within the default SMTP server stack.
> 
> ## IMAP
> 
> Implementationof RFC-3501 to enable a user to access and manage their
> mailbox.
> 
> A user as of today can not extend the behaviour of the IMAP stack.
> 
> ## JMAP
> 
> Implementation of RFC-8620 and RFC-8621 to enable a user to interact
> and
> manage their mailboxes, and to send emails, on top of HTTP.
> 
> The implementation of these specifications is currently a work in
> progress. A previous draft implementation is available.
> 
> ## POP3
> 
> Implementation of RFC-1939 to enables users to retrieve the mails
> within
> their mailboxes.
> 
> A user can register additional **ProtocolHandlers** to extend the
> capabilities of the POP3 server.
> 
> ## ManagedSieve
> 
> Implementation of RFC-5804 to enable a user to upload and manage
> their
> Sieve scripts.
> 
> This protocol implementation is known to be buggy.
> 
> # Server Administration
> 
> ## REST administration via WebAdmin
> 
> James specific REST APIs to manage other components, and their
> specific
> implementations.
> 
> WebAdmin routes are defined in a modular way, the webadmin API will
> thus
> be product specific.
> 
> Specification:
> 
> Key interface:
>  - **Routes** enables registering additional REST endpoints.
> 
> Plugin: a user can define custom webadmin routes.
> 
> ## Administration via the CLI
> 
> Command Line interface to manage James.
> 
> This CLI is not product specific, some command will not work for some
> products.
> 
> The CLI relies on JMX protocol and can represent a potential
> vulnerabilities. Guice products enables disabing it. A long term work
> is
> to port the CLI to rely on the WebAdmin REST APIs.
> 
> # Utility components
> 
> These components offers services core-components implementations can
> rely on.
> 
> ## BlobStore
> 
> Stores potentially large binary data.
> 
> Entity:
>  - **Blob**: Binary data
> 
> Key interface: BlobStore
> 
> Implementations:
>  - Memory
>  - ObjectStorage (S3/Swift)
> 
> ## TaskManager
> 
> Allows to control and schedule long running tasks run by other
> components. Among other it enables scheduling, progress monitoring,
> cancelation of long running tasks
> 
> Entity:
>  - **Task**: An operation performed by the TaskManager
>  - **Task additional information**: Task specific information.
> Exposes
> specific details about this task and how its execution went on.
> 
> Key interface: **TaskManager**.
> 
> Implementations:
>  - Memory
>  - Distributed (cassandra + RabbitMQ)
> 
> ## Metrics
> 
> Enables recording execution timing for various operation within
> James.
> Enables fine grained performance monitoring of a running James
> server,
> for example using grafana boards.
> 
> Key interface: **MetricFactory**.
> 
> Implementation:
>  - Default
>  - DropWizard
> 
> ## HealthChecks
> 
> Enables knowing the status of each components within a running James
> server.
> 
> This is both periodically logged, and exposed via WebAdmin.
> 
> Key interface: **HealthCheck**.
> 
> ## Event sourcing
> 
> Event sourcing implementation for the James server. Enables
> components
> to rely on event sourcing technics for taking decisions.
> 
> Here the definition of event is not the one of the mailbox, we are in
> a
> different bounded context.
> 
> Entity:
> 
>  - **Aggregate** represents an entity on which actions are performed
>  - **command** represents actions we wish to perform on this
> aggregate
>  - **events** represent modifications of the aggregate. The state of
> the
> aggregate can be computed from its event history.
>  - **CommandHandler** applies commands on aggregates and generates
> the
> resulting events.
>  - **Subscriber** enables applying actions upon events.
> 
> Key interfaces:
> 
>  - **EventStore** stores events generated by event sourcing.
> Implemented
> in Memory and on top of Cassandra.
> 
> ## JSON serialization
> 
> Generic mechanism for modular JSON serialization of entities.
> 
> Sample usages:
>  - Task
>  - Task additional details
>  - Event sourcing events
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

James architecture: core components, sub-components and utility components

[Tellier Benoit <bt...@apache.org>]

Hi David,

I gave a shot at defining:

 - Core components, that offers services at heart of James
 - Utility components, that offers services core-components
implementations can rely on.
 - Mailbox sub-components.

Here is the result. I encourage other project members to review it.

 - Did I forgot some components?
 - Is the way I split them relevant?
 - Is the way to describe a component (descriptions, entities, key
interfaces, plugin) relevant? Would you add/remove something?
 - Of course all the component level details...

I hope it helps your documentation effort.

Best regards,

Benoit

-------------------------------------

# Core components

## MailetContainer

Enables mail processing.

Entity:
 - **Mails**: Message transiting between a sender and recipients

Key interfaces:
 - **Mailet**: Enables to act upon an email: modify it, trigger side effects
 - **Matcher**: Condition to trigger a mailet action

Plugin: A user can register their own mailets / matchers

Here is a list of mailets providing central services, and bridges other
core components defined here after :

 - **RemoteDelivery** enables sending emails to other mail servers,
using the SMTP protocol.
 - **LocalDelivery** enables delivering emails to local user mailboxes.
 - **RecipientRewriteTable** queries and applied recipient rewritting rules.
 - **ToMailRepository** stores emails in a given mail repository.
 - **Sieve** executes stored sieve scripts against incoming emails.

## MailQueue

Enables asynchronous mail processing. Enables review and administration
of mail traffic awaiting processing.

Entity: **Mails**

Key interface: **MailQueue**, **ManageableMailQueue**

Implementations:
 - Memory
 - (embedded) ActiveMQ
 - Distributed (RabbitMQ + Cassandra)
 - Files

## MailRepositories

Enables storing emails, along with their processing context. Enables
traffic review.

Can be used to recover error, stores spam, etc...

Entity: **Mails**

Key interfaces:
 - **MailRepository**
 - **MailRepositoryStore**: enables instanciation of MailRepositories

Plugins:
 - A user can register new classes of MailRepository, that he can then
configure within it's Mailet Container

Bundled implementation:
 - Memory
 - Cassandra
 - JDBC
 - File

## Mailbox

The mailbox stores user messages for later retrieval.

Entities:
 - **Messages**: a mime message belonging to a user, along with their
mailbox context.
 - **Mailboxes**: a group of messages

Sample operations:
 - Mailboxes can be created, deleted, renamed
 - Messages can be appended, deleted, moved, copied, their flags can be
modified

Key interfaces:
 - **MailboxManager**: Enable managing and accessing mailboxes
 - **MessageManager**: Enables accessing messages within a given
mailbox. Can be obtained via the MailboxManager.
 - **MessageIdManager**: Enables accessing messages by their unique
identifier.. Can be obtained via the MailboxManager.

Implementations:
 - Memory implementation
 - Maildir implementation
 - JPA implementation
 - Cassandra implementation

The mailbox components defines the following sub-components:

### The EventBus

James mailbox uses an event driven architecture.
	
It means every meaningful action on mailboxes or messages triggers an
event for any component to react to that event.
`MailboxListener` allows executing actions upon mailbox events. They
could be used for a wide variety of purposes, like
enriching mailbox managers features or enabling user notifications upon
concurrent mailboxes operations.

Entities:
 - **Events**: describes an action that happenened on a mailbox
 - **MailboxListeners**: standard API for acting upon events.

Key interfaces:
 - **EventBus**: enables MailboxListener registration and enables
dispatching events to the MailboxListeners

Implementations:
 - In VM
 - RabbitMQ

Plugins:
 - A user can register its own mailbox listeners.

### EventDeadLetter

Failed event processing is being saved to the EventDeadLetter for both
diagnostic and reprocessing purposes.

Entities:
 - **Events**: As per the EventBus

Key interfaces:
 - **EventDeadLetter**

Implementations:
 - Cassandra
 - Memory

### Search index

Sub components allowing searching emails.

Entities: **Messages**

Key interfaces:
 - **MessageSearchIndex**

Implementations:
 - Scrolling
 - Lucene
 - ElasticSearch

### Text extraction

Allows extracting text from arbtrary files. It empowers a more relevant
search within the MessageSearchIndex

Key interfaces:
 - **TextExtractor**

Implementations:
 - Default: only extracts plain text
 - JSoup: only extracts plain text or HTML
 - Tika: relias on Apache Tika

### Quotas

Defines and enforces limitations on the mailbox resource usage. Keep
track of current resource usage.

Entities:
 - Quota Root: defines a group of mailboxes for which a given quota
applies. James implementation defines QuotaRoot as the mailboxes
belonging to a user.
 - Resources: What resourc ehte quota tracks. Could be the count of
messages or the total sizes of messages.
 - Quota: A limit along with current usage for a given Quota Root.

Key interfaces:
 - **QuotaManager** allows to define limits and retrieve quota usage.

## data/api

Stores server level metadata.

Entities:
 - **Users**: people enables to access the server, along with their
credentials
 - **Domains**: Logical group of users

Key interfaces:
 - **UsersRepository**
 - **DomainList**

Implementation:
 - Memory
 - Cassandra
 - JPA

## Recipient Rewrite Tables

Enables storing rules for rewriting recipient of an email.

Entities:
 - **Mapping source**: defines which address should be rewritten for a
given rule
 - **Mapping**: defines the rewriting that should be performed

Key interface: **RecipientRewriteTable** stores all the Mappings along
with their sources.

Implementations:
 - Memory
 - Cassandra
 - JPA
 - XML (configuration only)

## SieveRepository

Enables users to store Sieve scripts. Enables managing quota applied for
user Sieve scripts.

Entity:
 - **Sieve script**: enable a user to define actions to be performed
upon mail reception.

Key interfaces:
 - **SieveRepository**: Leverages sieve script storage
 - **SieveQuotaRepository**: Leverages sieve script quota storage

## DNS Service

Provides abstraction for DNS resolutions. Helps DNS resolution for mail
processing purposes.

Key interfaces: **DNSService** enables DNS resolution.

# Protocols

## SMTP

Implementation of RFC-5321 to enable receiving emails. Mails are
processed asynchronously.

A user can register additional **ProtocolHandlers** to extend the
capabilities of the SMTP server.

A SMTP **Hook** is a specific Protocol handler being plugged in a
specific location within the default SMTP server stack.

## LMTP

Implementation of RFC-2033 to enable receiving emails. Similar to SMTP
but mails are processed synchronously.

A user can register additional **ProtocolHandlers** to extend the
capabilities of the LMTP server.

A SMTP **Hook** is a specific Protocol handler being plugged in a
specific location within the default SMTP server stack.

## IMAP

Implementationof RFC-3501 to enable a user to access and manage their
mailbox.

A user as of today can not extend the behaviour of the IMAP stack.

## JMAP

Implementation of RFC-8620 and RFC-8621 to enable a user to interact and
manage their mailboxes, and to send emails, on top of HTTP.

The implementation of these specifications is currently a work in
progress. A previous draft implementation is available.

## POP3

Implementation of RFC-1939 to enables users to retrieve the mails within
their mailboxes.

A user can register additional **ProtocolHandlers** to extend the
capabilities of the POP3 server.

## ManagedSieve

Implementation of RFC-5804 to enable a user to upload and manage their
Sieve scripts.

This protocol implementation is known to be buggy.

# Server Administration

## REST administration via WebAdmin

James specific REST APIs to manage other components, and their specific
implementations.

WebAdmin routes are defined in a modular way, the webadmin API will thus
be product specific.

Specification:

Key interface:
 - **Routes** enables registering additional REST endpoints.

Plugin: a user can define custom webadmin routes.

## Administration via the CLI

Command Line interface to manage James.

This CLI is not product specific, some command will not work for some
products.

The CLI relies on JMX protocol and can represent a potential
vulnerabilities. Guice products enables disabing it. A long term work is
to port the CLI to rely on the WebAdmin REST APIs.

# Utility components

These components offers services core-components implementations can
rely on.

## BlobStore

Stores potentially large binary data.

Entity:
 - **Blob**: Binary data

Key interface: BlobStore

Implementations:
 - Memory
 - ObjectStorage (S3/Swift)

## TaskManager

Allows to control and schedule long running tasks run by other
components. Among other it enables scheduling, progress monitoring,
cancelation of long running tasks

Entity:
 - **Task**: An operation performed by the TaskManager
 - **Task additional information**: Task specific information. Exposes
specific details about this task and how its execution went on.

Key interface: **TaskManager**.

Implementations:
 - Memory
 - Distributed (cassandra + RabbitMQ)

## Metrics

Enables recording execution timing for various operation within James.
Enables fine grained performance monitoring of a running James server,
for example using grafana boards.

Key interface: **MetricFactory**.

Implementation:
 - Default
 - DropWizard

## HealthChecks

Enables knowing the status of each components within a running James server.

This is both periodically logged, and exposed via WebAdmin.

Key interface: **HealthCheck**.

## Event sourcing

Event sourcing implementation for the James server. Enables components
to rely on event sourcing technics for taking decisions.

Here the definition of event is not the one of the mailbox, we are in a
different bounded context.

Entity:

 - **Aggregate** represents an entity on which actions are performed
 - **command** represents actions we wish to perform on this aggregate
 - **events** represent modifications of the aggregate. The state of the
aggregate can be computed from its event history.
 - **CommandHandler** applies commands on aggregates and generates the
resulting events.
 - **Subscriber** enables applying actions upon events.

Key interfaces:

 - **EventStore** stores events generated by event sourcing. Implemented
in Memory and on top of Cassandra.

## JSON serialization

Generic mechanism for modular JSON serialization of entities.

Sample usages:
 - Task
 - Task additional details
 - Event sourcing events

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Hexagonal Architecture - definitions of fundamental concepts

[David Leangen <ap...@leangen.net>]

>> Thanks Benoit, very helpful! It does lead to more questions, though. :-)
> 
> The contrary would have surprised me ;-)

Thanks again! There is a lot to digest here. All this information has been really useful.

I will try to put my thoughts together by advancing with the documentation before I come back. Likely I’ll start new threads that are more fine-grained, since there is so much going on in this one thread.


Cheers,
=David

Re: Hexagonal Architecture - definitions of fundamental concepts

[Tellier Benoit <bt...@apache.org>]

On 22/05/2020 18:06, David Leangen wrote:
> Thanks Benoit, very helpful! It does lead to more questions, though. :-)

The contrary would have surprised me ;-)

I have been yet again quitte verbose once again, don't hesitate to raise
a warning if the signal/noise ratio gets to low...

Cheers,

> 
>> - **mailet** are separated in an effort to serve as a specification for...
>> - **mailbox** as presented below is a storage service for the server….
>> - **protocols** is an effort to provide mail protocol utility for...
>> I hope I provided some context about the project structure…
>> I would call the core component the **mailet container**...
> 
> 
> Yes, indeed! Let me try to rephrase this in my own wording to test if I understood. I’ll use the Servlet spec as an analogy, because what you write about Mailet and protocols triggered that comparison in my mind.
> 
> If we use the analogy of the HTTP<—>Servlets, then in this case there would be a similar relationship between for example IMAP<—>Mailet.
> 
> A gross simplification of the essence of a generic Servlet is:
> 
> ```
> public interface Servlet {
>   void service(Request req, Response rsp) throws ServletException;
> }
> ```
> 
> More specific to the HTTP protocol is:
> 
> ```
> public class HttpServlet {
>   public void get(HttpRequest req, HttpResponse rsp) throws ServletException;
>   public void put(HttpRequest req, HttpResponse rsp) throws ServletException;
>   public void post(HttpRequest req, HttpResponse rsp) throws ServletException;
>   public void delete(HttpRequest req, HttpResponse rsp) throws ServletException;
>   public void etc(HttpRequest req, HttpResponse rsp) throws ServletException;
> }
> ```
> 
> 
> So for a Mailet, the idea would be:
> 
> ```
> public interface Mailet {
>   void service(Request req, Response rsp) throws MailetException;
> }
> ```
> 
> And for the IMAP protocol:
> 
> ```
> public class ImapMailet {
>   public void doSomethingImapSpecific(IMAPRequest req, IMAPResponse rsp) throws MailetException;
> }
> ```

Mailet is intended for a specific goal: Mail processing.

Regarding IMAP, the components processing IMAP commands are ImapProcessor.

> 
> However, the actual generic signature is:
> 
> ```
> void service(Mail mail) throws MessagingException;
> ```
> 
> (By the way, does this mean that the Mail is mutable and that the service method creates a side-effect by modifying the Mail object??)

Yes.

(To be honest, in the comparison you do a servlet is doing a side effect
to send a response).

That 'mutation' is due to the lack of efficient immutable data-structure
to represent a mime object (long term goal of the mime4j library).

> 
> Is the analogy something along the right lines? So for example: "Tomcat is to the Servlet spec as James is to the Mailet spec"??
> 
> 
> Even if the analogy makes sense, the one big difference is that Servlets are stateless, but emails are not, so as you pointed out it needs the concept of Mailboxes for storage. 

There is a lot to say about this sentence.

Most (all?) mailets in James are stateless.

The mailbox is needed even with an immutable processing. As it
represents the mails that a given user received, so that he can read
them when he wishes.

The "storing an email in a mailbox" is a business rule, not a technical
limitation.

Note that mailbox is not a hard dependency for the mailet-container. It
is defined via a plugin (a LocalDelivery mailet) and you can very well
introduce a profile running completly without it. We did it in a SMTP
only James profile tested here [1] (not exposed to end user, but rather
intendeed to demonstrate James as a toolkit to build email server)


[1]
https://github.com/apache/james-project/blob/master/mpt/impl/smtp/cassandra-rabbitmq-object-storage/src/test/java/org/apache/james/mpt/smtp/CassandraRabbitMQAwsS3SmtpTestRuleFactory.java

> 
> 
>>> * Mime4j
>>> * jSieve
>>> * jSFP
>>> * jDKIM
>>
>> I would call them "external librairies".
> 
> To me, according to the vocabulary that I am used to (meaning that I am not claiming to be “right”, just that this is my experience), a "library" is something that gets embedded into a "component". Both are just ways of packaging up code, but the library is both “hidden” and "disposable" (meaning that it is just part of the implementation and is not seem from the outside, nor cared about), while the "component" is something that is actually seen from the outside and worth understanding in order to understand the entire system.
> 
> If users or maybe even developers of James don't have to care about Mime4j and it can be hidden away, then I would agree that it is a library.

So far I would agree with that statement.

> It would just be part of some other components, and would only come into scope when developing that component. So for this documentation project, I probably wouldn't even have to mention it at all.
> 
> Or in other words, only one of the following is correct:
> 
>  * These concepts are understood internally by James core, but the implementation code is provided by these libraries and does not need to be exposed to the outside
> 
> OR
> 
>  * These concepts are not understood by the James core, so both the concepts (API) and the implementation are provided by these external projects

Sorry if I did not expressed myself correctly

By "library" I mean that these projects are used outside their James
context.

They offer a common service that java projects working with emails might
be interested in.

See for instance this reaction:
http://mail-archives.apache.org/mod_mbox/james-mime4j-dev/202005.mbox/%3C16B1D214-B4E6-4230-96AC-CEDCD2D0D934%40whiuk.com%3E


This might be seen also as an effort to "advertise" the James ecosystem.
> 
> 
>>> Administration
>>> ------------------------------------
>>> The port is the “Admin API”. There are 3 adapters: JMX, REST, and CLI.
>>
>> A user could be loading additional webadmin routes (~plugin)
> 
> Sorry, I don’t understand this statement…

Sorry.

As a user running James, I can write my own custom WebAdmin API routes
and load them into a running James server.

This will result in these routes being expose within that James server.

So "webadmin routes" is a plugin, a "port" a developer can use to modify
James behaviour (somehow just like mailets)

> 
> 
> 
>>> Storage API
>>> ------------------------------------
>>> This is actually a general term for 3 APIs:
>>>
>>> * Mailbox API
>>> * Search API
>>
>> Which is called "search API" is in my opinion a sub-part of the
>> mailbox-api.
> 
> Ok. As for the rest of of the details you provided about the mailbox-api, I’ll get back to that later.
> 
> 
>>
>>> * User API
>>
>> * data-api allow persistence of server data like users (above
>> mentionned User API), domains, recipient rewriting rules. There might be
>> some work to better name this component, and refine its boundaries.
>>
>> There is I think much more going on at the storage level.
> 
> But you already wrote above that it is Mailbox that describes storage in the context of a Mailet-based system like James.

No.

mailbox-api is loosely related to mailet and mail processing.

Storing a "message" in a mailbox is a potential result of processing a mail.

Again the entities needs to be more clearly defined.

 - A mail is a MIME message in transit (with a sender and recipients)
 - A message belongs to a user, who can read it, retrieve it, delete it,
mark it as read, etc.

A mailbox stores message, not mails.

> 
> If something is being stored, then it must be because the concept is known to the core part of the system.

Yes.

The LocalDelivery mailet is acting as an adapter between the
mailet-container component (where emails flows) and the mailbox component.

A nice diagram might be http://james.apache.org/images/james-imap-server.png

> Or are you saying that the core offers a generic “data storage” service to outside components that want to use it? So it is some generic storage service like labels in Kubernetes that is used however external libraries want to use it?
> 
> 
>> * mailQueue is clearly a central component is James (as in any SMTP server)
> 
> But I have understood from all of the documentation and our emails so far that "SMTP is just a protocol". It sounds like there is more to it than just a protocol, then. There are expectations about behaviour and the services offered by the system.

SMTP defines the fact that emails gets asynchronously processed. If not,
you have the "local" mail transfer protcol (LMTP)

> 
> 
>> * blob-store is a component for storing potentially large binary data.
>> Other core-components implementations can rely on it to store
>> (potentially large) binary data
> 
> How is this related to Mailbox? Does Mailbox use the blob store (so Mailbox is like the Aggregate Root if this were a DDD concept)? Or does Mailbox get bypassed, so that actually storage is more like Mailbox+blobstore?

A mailbox implementation *might* use the blobStore component.

The cassandra mailbox does this, not the JPA one.

(Blob Store is not a top level component but rather a utility other
components can use)

> 
> 
>> * mail-repository allows storage of email with their processing context
> 
> So you mean it adds extended functionality to a Mailbox? I’m not entirely sure I understand this statement.

The entity is not the same.

Mailbox stores messages belonging to a user, who can read it, retrieve
it, delete it, mark it as read, etc.

MailRespository stored mails (with their envelope) that we don't want to
flow anymore. It could be used upon error, spam, etc... An administrator
can reprocess them anytime he wishes.

> 
> 
>> * sieve script storage (nowadays part of data-api)
> 
> Right, but I already understood that sieve is “external”, so now I am confused.

JSieve is a library helping with parsing Sieve scripts.

It has some external usages (Zimbra), and lives so far as a standalong
library.

Now in James we have the following features related to Sieve:

 - As a user I want to store sieve scripts on the server
 - As a user I want my Sieve scripts to be executed upon mail reception

So we need some extra components in James in addition to the Jsieve
library to make this work:

 - The webAdmin routes to manage Sieve scripts
 - The Sieve management mailet (ManageSieveMailet)
 - The managesieve protocol (experimental)
 - Sieve script storage.

These services goes way further than just the Sieve script parsing
offered by JSieve.

Clearer?

> 
> 
> 
>> Task manager
>> ---------------------
>> Allows to control and schedule long running tasks run by other components.
>> We should, in my opinion, allow additional task registration via plugins.
> 
> So there is a core concept of a “task”? Is this a cross-cutting concern that is known by all components of the system? So for example the Mailbox admin knows that “Move Mailbox” is a task, and therefore it can be scheduled?
> 
> Can you please explain more about what a “task” is in James, and how it is used?

Same than blobStore. It is more of a utility that other components might
rely on.

To give you some example:
 - Re-processing some emails stored in a mail repository is a task.
 - Removing some mails in the mail queue is a task.
 - Migrating data between two Cassandra tables is a task.

All these core-components have similar needs (cancellation, monitoring
of long running jobs) that the taskManager helps to fullfill.
> 
> 
> Cheers

As a conclusion, we might rely benefit from documenting the core
entities core components acts on, I think it could clearly ease your
understanding.

Also we should focus on core components first, utility components
(blobStore, taskManager) are likely less useful for the big picture
understanding. Sorry for the digression.

> =David
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Hexagonal Architecture - definitions of fundamental concepts

[David Leangen <ap...@leangen.net>]

Thanks Benoit, very helpful! It does lead to more questions, though. :-)


> - **mailet** are separated in an effort to serve as a specification for...
> - **mailbox** as presented below is a storage service for the server….
> - **protocols** is an effort to provide mail protocol utility for...
> I hope I provided some context about the project structure…
> I would call the core component the **mailet container**...


Yes, indeed! Let me try to rephrase this in my own wording to test if I understood. I’ll use the Servlet spec as an analogy, because what you write about Mailet and protocols triggered that comparison in my mind.

If we use the analogy of the HTTP<—>Servlets, then in this case there would be a similar relationship between for example IMAP<—>Mailet.

A gross simplification of the essence of a generic Servlet is:

```
public interface Servlet {
  void service(Request req, Response rsp) throws ServletException;
}
```

More specific to the HTTP protocol is:

```
public class HttpServlet {
  public void get(HttpRequest req, HttpResponse rsp) throws ServletException;
  public void put(HttpRequest req, HttpResponse rsp) throws ServletException;
  public void post(HttpRequest req, HttpResponse rsp) throws ServletException;
  public void delete(HttpRequest req, HttpResponse rsp) throws ServletException;
  public void etc(HttpRequest req, HttpResponse rsp) throws ServletException;
}
```


So for a Mailet, the idea would be:

```
public interface Mailet {
  void service(Request req, Response rsp) throws MailetException;
}
```

And for the IMAP protocol:

```
public class ImapMailet {
  public void doSomethingImapSpecific(IMAPRequest req, IMAPResponse rsp) throws MailetException;
}
```

However, the actual generic signature is:

```
void service(Mail mail) throws MessagingException;
```

(By the way, does this mean that the Mail is mutable and that the service method creates a side-effect by modifying the Mail object??)

Is the analogy something along the right lines? So for example: "Tomcat is to the Servlet spec as James is to the Mailet spec"??


Even if the analogy makes sense, the one big difference is that Servlets are stateless, but emails are not, so as you pointed out it needs the concept of Mailboxes for storage. 


>> * Mime4j
>> * jSieve
>> * jSFP
>> * jDKIM
> 
> I would call them "external librairies".

To me, according to the vocabulary that I am used to (meaning that I am not claiming to be “right”, just that this is my experience), a "library" is something that gets embedded into a "component". Both are just ways of packaging up code, but the library is both “hidden” and "disposable" (meaning that it is just part of the implementation and is not seem from the outside, nor cared about), while the "component" is something that is actually seen from the outside and worth understanding in order to understand the entire system.

If users or maybe even developers of James don't have to care about Mime4j and it can be hidden away, then I would agree that it is a library. It would just be part of some other components, and would only come into scope when developing that component. So for this documentation project, I probably wouldn't even have to mention it at all.

Or in other words, only one of the following is correct:

 * These concepts are understood internally by James core, but the implementation code is provided by these libraries and does not need to be exposed to the outside

OR

 * These concepts are not understood by the James core, so both the concepts (API) and the implementation are provided by these external projects



>> Administration
>> ------------------------------------
>> The port is the “Admin API”. There are 3 adapters: JMX, REST, and CLI.
> 
> A user could be loading additional webadmin routes (~plugin)

Sorry, I don’t understand this statement…



>> Storage API
>> ------------------------------------
>> This is actually a general term for 3 APIs:
>> 
>> * Mailbox API
>> * Search API
> 
> Which is called "search API" is in my opinion a sub-part of the
> mailbox-api.

Ok. As for the rest of of the details you provided about the mailbox-api, I’ll get back to that later.


> 
>> * User API
> 
> * data-api allow persistence of server data like users (above
> mentionned User API), domains, recipient rewriting rules. There might be
> some work to better name this component, and refine its boundaries.
> 
> There is I think much more going on at the storage level.

But you already wrote above that it is Mailbox that describes storage in the context of a Mailet-based system like James.

If something is being stored, then it must be because the concept is known to the core part of the system. Or are you saying that the core offers a generic “data storage” service to outside components that want to use it? So it is some generic storage service like labels in Kubernetes that is used however external libraries want to use it?


> * mailQueue is clearly a central component is James (as in any SMTP server)

But I have understood from all of the documentation and our emails so far that "SMTP is just a protocol". It sounds like there is more to it than just a protocol, then. There are expectations about behaviour and the services offered by the system.


> * blob-store is a component for storing potentially large binary data.
> Other core-components implementations can rely on it to store
> (potentially large) binary data

How is this related to Mailbox? Does Mailbox use the blob store (so Mailbox is like the Aggregate Root if this were a DDD concept)? Or does Mailbox get bypassed, so that actually storage is more like Mailbox+blobstore?


> * mail-repository allows storage of email with their processing context

So you mean it adds extended functionality to a Mailbox? I’m not entirely sure I understand this statement.


> * sieve script storage (nowadays part of data-api)

Right, but I already understood that sieve is “external”, so now I am confused.



> Task manager
> ---------------------
> Allows to control and schedule long running tasks run by other components.
> We should, in my opinion, allow additional task registration via plugins.

So there is a core concept of a “task”? Is this a cross-cutting concern that is known by all components of the system? So for example the Mailbox admin knows that “Move Mailbox” is a task, and therefore it can be scheduled?

Can you please explain more about what a “task” is in James, and how it is used?


Cheers,
=David



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Hexagonal Architecture - definitions of fundamental concepts

[Tellier Benoit <bt...@apache.org>]

Hello David,

> I know you are all probably very busy, but to allow my contribution to
be useful, it would be very nice to get comments on my journey so far.
If I don’t document the “right” thing and run out of time, it would be a
wasted opportunity. :-(

100% agree with that.

Don't hesitate if you have questions on these topics.

I tried to help you by providing as much details as I could think of.

I believe what is important is defining:

 - Core components
 - Port for external devices
 - Port exposed to the user to extend James behavior, which I refer to
as "Pugins."

> One image I have seen is this one:
>
>   https://james.apache.org/images/james-schema-subprojects.png
>
> It shows that the “central projects” are:
>
>  * Server
>  * Mailets
>  * Mailbox
>  * Protocol
While here I agree with the wording (project structure is mostly history
related) it do partially reflect "system architecture".

 - **mailet** are separated in an effort to serve as a specification for
mail processing. And "James" would tend to be "one of the implementation
for running mailets". Maybe a question as a project would be to
re-evaluate that goal: are mailets just a port for the Apache James
server or used by other projects as a mail processing specification?

 - **mailbox** as presented below is a storage service for the server.
Would it make sense out of the James server?

 - **protocols** is an effort to provide mail protocol utility for
parsing, without hard dependency on server components, and without
trasport implementation. **server/protocols** adapts **protocols** to
the James components, relying on **netty** for transport.

Stepping back, that goal is achieved for SMTP, however IMAP, POP3 and
the like depends on "mailbox/api".

I hope I provided some context about the project structure, and why it
do not match the server architecture. Re-evaluating some of the above
mentionned goals could be an interesting milestone for this PMC.

> I am assuming that “central projects” is synonymous with “core”
projects. So according to this diagramme, those projects are “core”
projects.
>
> Other “important” projects are:
>
>  * Mime4j
>  * jSieve
>  * jSFP
>  * jDKIM
>
> However, I will **not** address these yet, so I will completely ignore
them for now. According to the image, they are clearly *not* core. They
are noted as being “External Projects”.

True.

I would call them "external librairies".


[...]

> Mailet Containers
> ------------------------------------
> Each Mailet is in itself an adapter, while the Mailet “specification” is the port. The port is the interface/contract that is known and understood by both the core and the outside world. Core doesn’t care what the Mailet does, so long as each Mailet respects the contract, which is defined by the Mailet “specification”.
> 
> This is the main way that outside users can customize the behaviour of the system.

:+1:

I would call the core component the **mailet container**.

> 
> 
> Protocols
> ------------------------------------
> Each protocol is also a port, and is known by both the core and the outside. The shared knowledge is the external specification.
> 
> Since these specifications are well-known, there are many “External Devices” (applications) that implemented them.

The SMTP protocols, LMTP protocol and POP3 protocols allow a user to
load additional **ProtocolHandler** plugins, extending the behaviour of
these protocols.

> 
> 
> Administration
> ------------------------------------
> The port is the “Admin API”. There are 3 adapters: JMX, REST, and CLI.

A user could be loading additional webadmin routes (~plugin)

> 
> 
> Storage API
> ------------------------------------
> This is actually a general term for 3 APIs:
> 
>  * Mailbox API
>  * Search API

Which is called "search API" is in my opinion a sub-part of the
mailbox-api. That it is implemented via a sub-component is in my opinion
an implementation detail of the mailbox component.

The mailbox-api defines a plugin: the **mailbox listeners**. It allows
to trigger asynchronous actions upon mailbox event.

Also the **pre-deletion hooks** is a plugin that gets synchronously
called before deletion happens, allowing extra actions to be taken.

>  * User API

 * data-api allow persistence of server data like users (above
mentionned User API), domains, recipient rewriting rules. There might be
some work to better name this component, and refine its boundaries.

There is I think much more going on at the storage level.

 * mailQueue is clearly a central component is James (as in any SMTP server)

 * blob-store is a component for storing potentially large binary data.
Other core-components implementations can rely on it to store
(potentially large) binary data

 * mail-repository allows storage of email with their processing context

 * sieve script storage (nowadays part of data-api)

Task manager
---------------------

Allows to control and schedule long running tasks run by other components.

We should, in my opinion, allow additional task registration via plugins.

> Each implementation has an adapters for: Cassandra, MySQL, ElasticSearch
> 
> 
> I will proceed with this, and hope to hear comments soon.

I hope I did not loose myself too much into details, and again, don't
hesitate if you have questions!

> 
> 
> Thanks!!
> =David
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Hexagonal Architecture - definitions of fundamental concepts

[David Leangen <ap...@leangen.net>]

Hello,

> If we can reach agreement on a definition for each of these, then we can determine (1) how to document the concepts, and (2) if the code should be refactored to better reflect these concepts.


Since I am starting to run out of time, I need to press forward. So for now since I have not heard any objections so far to my proposal, I will continue with the understanding that my definitions of “Core, “Port”, “Adapter”, and “External Device” are acceptable. These are **not** core domain concepts, but I am using them so that I can understand what the core concepts are, and because as Matthieu wrote James is based on this architecture in this thread: https://www.mail-archive.com/server-dev@james.apache.org/msg65809.html.

To try to understand what the Core domain concepts are, I initially thought Guice could be useful, but as Matthieu pointed out in that thread, that is not the ideal approach. This time I will use the top-level documentation and try to match it with the Maven modules.

One image I have seen is this one: 

  https://james.apache.org/images/james-schema-subprojects.png

It shows that the “central projects” are:

 * Server
 * Mailets
 * Mailbox
 * Protocols

I am assuming that “central projects” is synonymous with “core” projects. So according to this diagramme, those projects are “core” projects.

Other “important” projects are:

 * Mime4j
 * jSieve
 * jSFP
 * jDKIM

However, I will **not** address these yet, so I will completely ignore them for now. According to the image, they are clearly *not* core. They are noted as being “External Projects”.

Another image is this one:

  https://james.apache.org/images/james-general-architecture.png

“James Core” is written in the middle. Although it is not immediately obvious, my interpretation is that the orange box represents the core (Server, Mailtets, Mailbox, Protocols), and the rest around the outside are the ports/adapters, which include:

Mailet Containers
------------------------------------
Each Mailet is in itself an adapter, while the Mailet “specification” is the port. The port is the interface/contract that is known and understood by both the core and the outside world. Core doesn’t care what the Mailet does, so long as each Mailet respects the contract, which is defined by the Mailet “specification”.

This is the main way that outside users can customize the behaviour of the system.


Protocols
------------------------------------
Each protocol is also a port, and is known by both the core and the outside. The shared knowledge is the external specification.

Since these specifications are well-known, there are many “External Devices” (applications) that implemented them.


Administration
------------------------------------
The port is the “Admin API”. There are 3 adapters: JMX, REST, and CLI.


Storage API
------------------------------------
This is actually a general term for 3 APIs:

 * Mailbox API
 * Search API
 * User API

Each implementation has an adapters for: Cassandra, MySQL, ElasticSearch …


I know you are all probably very busy, but to allow my contribution to be useful, it would be very nice to get comments on my journey so far. If I don’t document the “right” thing and run out of time, it would be a wasted opportunity. :-(


I will proceed with this, and hope to hear comments soon.


Thanks!!
=David



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org