James SMTP Model

[David Leangen <ap...@leangen.net>]

Hello,

As I mentioned during our call, I need your help so I can continue to move forward with the documentation.

My first questions are about SMTP.


* How does James model SMTP?
* Where can I find SMTP related concepts in the code?
* What are the related modules?
* How would I build my own SMTP server?
* What other information is relevant to SMTP that I may want to include?


Thanks!
=David

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Hi Eugen,

You will find my answers inlined.

> 
> I believe you make the assumption that people use and want to use the
> full plethora of protocols we have in James.
> 
> Sort of "All or nothing" approach.
> 
> Why do you think it is so?
> 
> I would argue that we should make little assumptions about how people
> deploy their infrastructure.
> 
> Maybe I have an existing infrastructure that I know and trust for SMTP
> and I just want to use the IMAP server from James.
> 
> Examples:
> 
> 1. I use Amazon Simple Email Service (or Sendgrid or Mailgun) for
> sending emails and I just want to be able to customize the IMAP/JMAP
> experience and control how I store emails.
> 
> 2. I could use James SMTP as a Gateway for my existing IMAP / Webmail setup.
> 
> 3. I could use James SMTP as a Relay for my marketing SaaS platform.

We should of course have an SMTP only server to address these use cases.

https://github.com/apache/james-project/tree/master/server/container/guice/jpa-smtp
achieve it.

> 
> 
> But that is not the main argument for having them separate.
> 
> The main argument IMO is that the protocols are quite complex and having
> the ability to work on just one protocol at a time has significant
> advantages:
> 
> * Lower cognitive effort - I only need to care about this specific
> protocol and it's implementation details, not the other protocols.
> 
>   This is one huge benefit.
> 
> * Faster development feedback cycle - If I work on IMAP or SMTP I should
> get away with running the tests only for that part.
> 
> * Less risk - If I work and break SMTP - the other parts should work.
> With the current project structure - it can be hard to say.
> 
> * Easier to adopt by new people - If people care only about SMTP, LMTP
> or IMAP - they can contribute only to that. The entry barrier will be
> smoother than what we have now.
> 
> * Easier to compose and adapt for new and interesting projects - If we
> package them independently and together in a single product (Advanced
> Server) we have a showcase of how you can do that. It will be clear that
> they are meant to be used in both ways.
> 
> 
> Having them as separate projects may have the downside of integrating
> them together as integrating them requires some form of glue.
> 
> The good side is that we already pay for this integration effort now :).
> 
> We will have to do the effort of splitting them apart while also keeping
> the full version.
> 
> This is doable and while it will take time it's not a hard problem IMO.
> 

Regarding Mail Delivery servers, you end up supporting IMAP/JMAP + SMTP.

I believe we should be doing a better job at making things optional and
packaged separately.

I would love to see for instance POP3, ManagedSieve and many other
things as extensions, bundled in separated JARs, opt-in.

Shipping a small set of servers, with very limited features, each one
with their set of extensions.

I believe it achieves most of what you describes but without requiring
complex integration.

> 
> Our way of using James is not the only way.
> 
> 
>>> [...]
> 
>>> NOTE: Netty should be upgraded to the latest version, I can do that once
>>> we are done with Gradle.
>> +1 <3
>>
>> Likely a huge effort though...
> You questioning my determination ?! (joke)

Never ^^

<joke>I don't know which one is easier... Gradle? Or Netty?</joke>
> 
> In time I will change the universe :D .

No doubt about it too.

Regards,

Benoit

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

> I gonna try my best, given my time constraints :-)

Thank you!

This is all very good information, which allows me to peel one more layer of the onion.

Likely my next batch of questions will be in a separate email thread, as I think we have exhausted the concept of “James SMTP Model”.

In the meantime, I have some things to work with so I can push forward a little more.


Thanks a lot!

Cheers,
=David

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Le 08/07/2020 à 14:40, David Leangen a écrit :
> Sorry, I do have one more question in response to your email…
>
> You wrote:
>
>> In my opinions we should document "How to write hooks with the
>> protocols/smtp library", "How to plug such hooks into a running James
>> server"
>>
>> Then "How to write commands for the protocols/smtp library" (and how to
>> plug them in James)
> This is starting to make better sense to me now. Thanks for your patience.
>
> Moving on… on this page:
>
>   —> https://james.apache.org/server/feature-smtp-hooks.html
>
>
>> The James SMTP Server Component allows to easy write your own code which will get executed in the SMTP-Transaction. Thats a bit different then using a Mailet a.k.a Mailet-API.
>>
>> To customize your SMTP Server, you have a few interfaces which helps you to "hook-in" a specific SMTP Command. That means your class which implements the given interface(s) will get called after the SMTP-Command was parsed and depending on your implementation it will handle it.
>>
>> As your code will get executed before the mail was even accepted. This can help you in many ways, most times its used for rejecting SPAM/Junk within the SMTP-Dialog. But it can be used for other things too.
>>
>> Its up to you and your use case.
>>
>> But be aware as your code needs to get executed during the SMTP-Transaction it should not take to long to execute. As it will need to fit in before the timeout was hit which can be different on every mail server. But as a general rule as long as your code can get executed within 30 seconds it should be fine.
>
> There is even a list of hooks on this page:
>
>   —> https://james.apache.org/server/dev-provided-smtp-hooks.html
>
>
> However… Can you provide more examples to help me better understand why I should care about these hooks, and why I would want to consider using a hook instead of a Mailet? Wouldn’t it be simpler just to have a single extension mechanism? What is the value of having two different extension mechanisms?
That is a VERY nice question that needs to be clearly highlighted in the
documentation "given my use case, which extension mechanism should I use".

There's somehow an overlap between hooks and mailets but:

 - Mailet is *asynchronous*. SMTP clients will not be affected by
potentially long processing. Hooks are on the other hand synchronous,
the SMTP client needs to wait for them to happen.
 - Hooks are very cheap. The mail is not stored yet. This makes it
really useful to get rid of spam via IP filtering, MX hosts checks and
so on. Another use case would be DOS prevention.
 - Hook allow configuration of extra additional behaviors at the SMTP
layer like "Sender needs to be valid, etc...".

I believe however that this extension mechanism is less useful than
mailet / matchers.

Does it helps?
>
> Thanks!
>
> =David
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

Sorry, I do have one more question in response to your email…

You wrote:

> In my opinions we should document "How to write hooks with the
> protocols/smtp library", "How to plug such hooks into a running James
> server"
> 
> Then "How to write commands for the protocols/smtp library" (and how to
> plug them in James)

This is starting to make better sense to me now. Thanks for your patience.

Moving on… on this page:

  —> https://james.apache.org/server/feature-smtp-hooks.html


> The James SMTP Server Component allows to easy write your own code which will get executed in the SMTP-Transaction. Thats a bit different then using a Mailet a.k.a Mailet-API.
> 
> To customize your SMTP Server, you have a few interfaces which helps you to "hook-in" a specific SMTP Command. That means your class which implements the given interface(s) will get called after the SMTP-Command was parsed and depending on your implementation it will handle it.
> 
> As your code will get executed before the mail was even accepted. This can help you in many ways, most times its used for rejecting SPAM/Junk within the SMTP-Dialog. But it can be used for other things too.
> 
> Its up to you and your use case.
> 
> But be aware as your code needs to get executed during the SMTP-Transaction it should not take to long to execute. As it will need to fit in before the timeout was hit which can be different on every mail server. But as a general rule as long as your code can get executed within 30 seconds it should be fine.


There is even a list of hooks on this page:

  —> https://james.apache.org/server/dev-provided-smtp-hooks.html


However… Can you provide more examples to help me better understand why I should care about these hooks, and why I would want to consider using a hook instead of a Mailet? Wouldn’t it be simpler just to have a single extension mechanism? What is the value of having two different extension mechanisms?


Thanks!

=David

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Le 06/07/2020 à 14:12, David Leangen a écrit :
> Thanks Benoit.
>
> Now, since I already have your attention on this topic, please allow me to really push so I can reach some kind of resolution. The resolution will allow me to figure out how to move forward with several things. Since I have been a bit stuck the past few days (and growing frustrated a little 😅) I hope you can continue to help me resolve this.
I gonna try my best, given my time constraints :-)
> Let me keep the focus on the SMTP module, because I think we’ll likely be able to extrapolate much of the approach to the other modules as well.
>
> I think this is what we seem to agree on:
>
>  * Separation of API and implementation is generally a good thing (especially if we want clean components)
>  * The current API is a bit messy and could use some cleaning
>  * However, the value of making any changes is questionable at this time
>
>
> Here are also a few things I have discovered on this journey:
>
>  * Actually implementing RFC5321 as a Java API would be VERY challenging
>      - The spec is fairly complex
>      - The spec is very “messy” and not very well or precisely written IMO
>  * As you pointed out, there are some important elements in the James SMTP module, such as commands and hooks
>      - These ought to be somehow highlighted and improved in the API
+1
>  * You have mentioned more than once that it uses a hexagonal architecture
>      - I think this is important, but I still don’t know what to do about it
>
>
> So I guess my question is this: what should I do in the immediate term? As you and others have pointed out (and me also), the documents should show what James *is*, not what James *aspires to be*.
In my opinions we should document "How to write hooks with the
protocols/smtp library", "How to plug such hooks into a running James
server"

Then "How to write commands for the protocols/smtp library" (and how to
plug them in James)

Using protocols-smtp without the server/protocols/protocols-netty super
layer would be so far a non-goal to me, and I would likely not be able
to help you do this.
> So what is the SMTP module, and how should I describe it?
I would state `james-project/protocols/protocols-smtp` is a transport
agnostic implementation of (some of) the SMTP specifications. It is
customizable (hooks) and extensible (additional commands).

`james-project/server/protocols/protocols-smtp` adapts
`james-project/protocols/protocols-smtp` in the James context, providing
hooks interacting with James server components. It also provides a TCP
transport layer, effectively providing a SMTP server.

Does it answers your question?
> In the part of the docs entitled “James model”, if I were to write in all honesty I don’t feel that there is a clear model, or at least it is totally not clear (to me) from the API. James is an implementation of RFC5321, and basically makes no attempt to have a clear java API. Rather, the API is via JMX. (I will have to investigate further to see what the JMX view of the system is.)
Ouch. Please don't.

It's old, not clean, mostly unmaintained, and likely to be removed in
the (not that far) future - for security reasons because JMX. Don't
hesitate if you want details on this statement (maybe a separate thread?)

To 'operate' James we should prefer webadmin. That will mean providing
CLI tool on top of it for the less technical of our users.

What do you try to achieve? (I don't see the link making you go from
SMTP to JMX to be honnest)
> I could mention that there is a “plan” to create cleaner java apis.
>
> That’s about all I can think of right now. Would this approach be acceptable?
There's always a plan in my opinion to consider API enhancement
proposals :-)
>
>>>> Note that "standard SMTP commands" are bundled into a single
>>>> ProtocolHandler, not relying on it means you defines your very own SMTP
>>>> commands.
>>> Ok, these are the types of things that I am very interested in understanding better. Are there any docs anywhere to help me out?Not really
>> Not really.
>>
>> http://james.apache.org/server/dev-provided-smtp-hooks.html defines hook
>> you can rely in for the default implementation.
>>
>> https://github.com/apache/james-project/blob/master/src/site/xdoc/server/dev-extend-smtp-hook.xml
>> might be helpful too.
>>
>> Which is not much for "hooks" already. Given the part of the quote you
>> chose, that might not be what you are looking for...
>>
>> Regarding writing your own commands, there is even less.
>>
>> http://james.apache.org/protocols/index.html is too generic
>>
>> http://james.apache.org/protocols/smtp.html is blank
> Would it be useful for me to pursue this in an attempt to document the SMTP module? I could try, but again I’ll most likely need help.
>
> Or should I write something like what I mention above and just be done with it for now?
You proved us writing documentation (and software) needs to be performed
with a usage in mind.

I would upvote (and support given my available time) documenting the
extension mechanisms of the existing protocols-smtp project. (so hooks
and commands) Rationals: that's the most straightforward use case for
protocols-smtp, and the documentation work should not be too hard to
perform.

(I forgot to mention
https://github.com/chibenwa/james-blacklist/blob/master/blacklist-smtp/src/main/java/com/linagora/james/blacklist/smtp/NotInBlackListHook.java
is an example of how to define a Rcpt hook, and the rest of the project
shows how to load it in a guice james server)

Cheers,

Benoit


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

Thanks Benoit.

Now, since I already have your attention on this topic, please allow me to really push so I can reach some kind of resolution. The resolution will allow me to figure out how to move forward with several things. Since I have been a bit stuck the past few days (and growing frustrated a little 😅) I hope you can continue to help me resolve this.

Let me keep the focus on the SMTP module, because I think we’ll likely be able to extrapolate much of the approach to the other modules as well.

I think this is what we seem to agree on:

 * Separation of API and implementation is generally a good thing (especially if we want clean components)
 * The current API is a bit messy and could use some cleaning
 * However, the value of making any changes is questionable at this time


Here are also a few things I have discovered on this journey:

 * Actually implementing RFC5321 as a Java API would be VERY challenging
     - The spec is fairly complex
     - The spec is very “messy” and not very well or precisely written IMO
 * As you pointed out, there are some important elements in the James SMTP module, such as commands and hooks
     - These ought to be somehow highlighted and improved in the API
 * You have mentioned more than once that it uses a hexagonal architecture
     - I think this is important, but I still don’t know what to do about it


So I guess my question is this: what should I do in the immediate term? As you and others have pointed out (and me also), the documents should show what James *is*, not what James *aspires to be*.

So what is the SMTP module, and how should I describe it?

In the part of the docs entitled “James model”, if I were to write in all honesty I don’t feel that there is a clear model, or at least it is totally not clear (to me) from the API. James is an implementation of RFC5321, and basically makes no attempt to have a clear java API. Rather, the API is via JMX. (I will have to investigate further to see what the JMX view of the system is.)

I could mention that there is a “plan” to create cleaner java apis.

That’s about all I can think of right now. Would this approach be acceptable?


>>> Note that "standard SMTP commands" are bundled into a single
>>> ProtocolHandler, not relying on it means you defines your very own SMTP
>>> commands.
>> Ok, these are the types of things that I am very interested in understanding better. Are there any docs anywhere to help me out?Not really
> 
> Not really.
> 
> http://james.apache.org/server/dev-provided-smtp-hooks.html defines hook
> you can rely in for the default implementation.
> 
> https://github.com/apache/james-project/blob/master/src/site/xdoc/server/dev-extend-smtp-hook.xml
> might be helpful too.
> 
> Which is not much for "hooks" already. Given the part of the quote you
> chose, that might not be what you are looking for...
> 
> Regarding writing your own commands, there is even less.
> 
> http://james.apache.org/protocols/index.html is too generic
> 
> http://james.apache.org/protocols/smtp.html is blank

Would it be useful for me to pursue this in an attempt to document the SMTP module? I could try, but again I’ll most likely need help.

Or should I write something like what I mention above and just be done with it for now?


Cheers,
=David

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Le 06/07/2020 à 13:12, David Leangen a écrit :
> [...]
>> I believe people are interested by a working SMTP implementation where
>> they can "plug" there behaviors or add the commands they miss.
> I am beginning to think that we may have a fundamentally differing understanding of what an API is. It may be a good idea at some point to have a very clear mutual understanding as to what an “API” is to James.
>
> I say that because I would never have written the sentence you wrote. I would have written “a pluggable component”, but never a “pluggable implementation” because no consumers should ever have to plug anything into an implementation. So how does something get plugged? It’s **necessarily** via the API (while the framework should handle the wiring with the implementation, just like Guide does at system startup). Only the API should ever be exposed to consumers. Consumers should never require the implementation code to compile their own code and work with James. It should always be enough to just have the API jar (consisting almost entirely of interfaces), with NO implementation jar (consisting of implementation classes).
>
> If you need to plug something into the implementation, then by definition the implementation is also an API, but then it means that you are mixing implementation with API, which is not usually a good practice if you want to have a componentized system. Consumers necessarily only interact with an API, by definition.
>
>
> Another purpose of having a solid API is for versioning. Normally an implementation will continue to advance (bug fixes, optimizations etc.), but for consumers of the API, the code they depend on (i.e. the API only) should change as little as possible.
>
> So the API ought to:
>
>  * Communicate the intention VERY well
>  * Not include “extra” stuff that is not absolutely necessary
>  * Definitely not include any implementations
>  * Allow consumers to minimize dependencies (or at least not require additional transitive dependencies)
>  * Allow consumers to swap different implementations (including different implementation versions)
>  * Allow consumers to use more recent implementations without having to change anything in their own system
Globally +1
>
> Here are an article that expresses views about APIs that are similar to mine:
>
>  * https://developer.ibm.com/technologies/java/articles/api-design-practices-for-java/
>
> It only touches on the differences between an API “consumer” and “provider”, but it would probably be a good thing to bring that into the James vocabulary as well so we can have slightly richer exchanges regarding APIs. I think what you are referring to is that we need to consider how consumers use an API, not just providers, and I totally agree.
>> (but again we need to consider the costs - return over investment ratio)
> Always! Completely agree. That is why I mentioned that I was having second thoughts and was starting to think that maybe we could get by WITHOUT having a proper java api (and maybe only a REST api, for instance).
>
> In that scenario, consumers of James would ONLY interact either via a REST API or SMTP / IMAP etc. There would simply be no java API because nobody would program against James.
>
> I think that could work, but we ought to make our approach explicit to avoid confusion.
+1 to be more specific.

We should at least extract the API of what todays works well:
 - Extra commands
- Hooks (additional behaviors)

And extract an API achieving these goals from the protocol-smtp package.

> Note that "standard SMTP commands" are bundled into a single
>> ProtocolHandler, not relying on it means you defines your very own SMTP
>> commands.
> Ok, these are the types of things that I am very interested in understanding better. Are there any docs anywhere to help me out?Not really

Not really.

http://james.apache.org/server/dev-provided-smtp-hooks.html defines hook
you can rely in for the default implementation.

https://github.com/apache/james-project/blob/master/src/site/xdoc/server/dev-extend-smtp-hook.xml
might be helpful too.

Which is not much for "hooks" already. Given the part of the quote you
chose, that might not be what you are looking for...

Regarding writing your own commands, there is even less.

http://james.apache.org/protocols/index.html is too generic

http://james.apache.org/protocols/smtp.html is blank

> Thanks as always for your patience.
>
You are welcome, and I'm happy you raise such an opportunity to
re-qualify or document things!
>
> Cheers,
> =David
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

> I don't think people are interested by an API allowing them to implement
> SMTP themselves. (That is quite some work!)

LOL indeed, and I completely agree. But I don’t think that’s the point I’m trying to make.

Some organizations may want to create an implementation.

The James team may wish to create a new implementation in the future.

More likely, people may want to be able to interface with a component (which requires a well-defined API), or compose their own system as you point out below.


> I believe people are interested by a working SMTP implementation where
> they can "plug" there behaviors or add the commands they miss.

I am beginning to think that we may have a fundamentally differing understanding of what an API is. It may be a good idea at some point to have a very clear mutual understanding as to what an “API” is to James.

I say that because I would never have written the sentence you wrote. I would have written “a pluggable component”, but never a “pluggable implementation” because no consumers should ever have to plug anything into an implementation. So how does something get plugged? It’s **necessarily** via the API (while the framework should handle the wiring with the implementation, just like Guide does at system startup). Only the API should ever be exposed to consumers. Consumers should never require the implementation code to compile their own code and work with James. It should always be enough to just have the API jar (consisting almost entirely of interfaces), with NO implementation jar (consisting of implementation classes).

If you need to plug something into the implementation, then by definition the implementation is also an API, but then it means that you are mixing implementation with API, which is not usually a good practice if you want to have a componentized system. Consumers necessarily only interact with an API, by definition.


Another purpose of having a solid API is for versioning. Normally an implementation will continue to advance (bug fixes, optimizations etc.), but for consumers of the API, the code they depend on (i.e. the API only) should change as little as possible.

So the API ought to:

 * Communicate the intention VERY well
 * Not include “extra” stuff that is not absolutely necessary
 * Definitely not include any implementations
 * Allow consumers to minimize dependencies (or at least not require additional transitive dependencies)
 * Allow consumers to swap different implementations (including different implementation versions)
 * Allow consumers to use more recent implementations without having to change anything in their own system

Here are an article that expresses views about APIs that are similar to mine:

 * https://developer.ibm.com/technologies/java/articles/api-design-practices-for-java/

It only touches on the differences between an API “consumer” and “provider”, but it would probably be a good thing to bring that into the James vocabulary as well so we can have slightly richer exchanges regarding APIs. I think what you are referring to is that we need to consider how consumers use an API, not just providers, and I totally agree.



> (but again we need to consider the costs - return over investment ratio)

Always! Completely agree. That is why I mentioned that I was having second thoughts and was starting to think that maybe we could get by WITHOUT having a proper java api (and maybe only a REST api, for instance).

In that scenario, consumers of James would ONLY interact either via a REST API or SMTP / IMAP etc. There would simply be no java API because nobody would program against James.

I think that could work, but we ought to make our approach explicit to avoid confusion.


> Note that "standard SMTP commands" are bundled into a single
> ProtocolHandler, not relying on it means you defines your very own SMTP
> commands.

Ok, these are the types of things that I am very interested in understanding better. Are there any docs anywhere to help me out?


Thanks as always for your patience.

Cheers,
=David

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Le 06/07/2020 à 07:13, David Leangen a écrit :
> Hey Benoit,
>
>> Now, that being said, I believe we should always define a "purpose" for
>> an API, and here we mixes things in my opinion.
> Yes, very good point. I agree 100%
>
>
>
>> Are we defining an API for building any arbitrary SMTP server?
> My somewhat vague thought is: I get the impression that in the Java world Apache James has become a kind of reference implementation. It could be that I have just not found anything else, or that nothing else exists, but in any case James is “important” in the intersection of “email” and “JVM”.
I don't think people are interested by an API allowing them to implement
SMTP themselves. (That is quite some work!)

I believe people are interested by a working SMTP implementation where
they can "plug" there behaviors or add the commands they miss.
> I think we should take it upon ourselves to view James and take on the role as the “reference implementation” for the JVM.
>
> This means that we ought to produce java apis that are compliant to the specs, that anybody could implement if they wanted to as technology develops. It also implies that we need to give the API lots of thought as this would become a de-facto industry standard, at least in the JVM world. Heck, if we do a really great job maybe we can even inspire improvements to the email spec itself. We should aim high.
>
>
> Take the fictitious case of a new, awesome “super-nutty” technology that is even better than Netty, but works in a completely different way. I should be able to make my own super-nutty implementation of smtp based on the James reference API.

Globally +1, if it lives as a separate library, we need a more
declarative way off assembling things...

(but again we need to consider the costs - return over investment ratio)

>> Or are we shipping an SMTP implementation with APIs to add new commands
>> support ?
> I am no expert in the spect, but aren’t you referring to ESMTP? Or do you mean something else?
>
> Even if we make a “reference api for Java”, there is nothing from stopping us from making an extension mechanism.
That mechanism already exists: one can register its own "protocol
handlers", be them extra commands or hooks into existing commands.

Note that "standard SMTP commands" are bundled into a single
ProtocolHandler, not relying on it means you defines your very own SMTP
commands.

Cheers,

Benoit


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

Hey Benoit,

> Now, that being said, I believe we should always define a "purpose" for
> an API, and here we mixes things in my opinion.

Yes, very good point. I agree 100%



> Are we defining an API for building any arbitrary SMTP server?

My somewhat vague thought is: I get the impression that in the Java world Apache James has become a kind of reference implementation. It could be that I have just not found anything else, or that nothing else exists, but in any case James is “important” in the intersection of “email” and “JVM”.

I think we should take it upon ourselves to view James and take on the role as the “reference implementation” for the JVM.

This means that we ought to produce java apis that are compliant to the specs, that anybody could implement if they wanted to as technology develops. It also implies that we need to give the API lots of thought as this would become a de-facto industry standard, at least in the JVM world. Heck, if we do a really great job maybe we can even inspire improvements to the email spec itself. We should aim high.


Take the fictitious case of a new, awesome “super-nutty” technology that is even better than Netty, but works in a completely different way. I should be able to make my own super-nutty implementation of smtp based on the James reference API.



> Or are we shipping an SMTP implementation with APIs to add new commands
> support ?

I am no expert in the spect, but aren’t you referring to ESMTP? Or do you mean something else?

Even if we make a “reference api for Java”, there is nothing from stopping us from making an extension mechanism.


Cheers,
=David

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Hi David,

As I exposed it earlier, protocols-smtp mixes the protocol
implementation and the interface it defines. We would benefit from
separating the too.

Now, that being said, I believe we should always define a "purpose" for
an API, and here we mixes things in my opinion.

Are we defining an API for building any arbitrary SMTP server?

Or are we shipping an SMTP implementation with APIs to add new commands
support ?

Or are we shipping an SMTP implementation with APIs to add new behavior
on existing commands ?

I believe protocols-smtp addresses the later two, and should be
documented/re-organised if needed with that optic in mind.

Best regards,

Benoit

Le 05/07/2020 à 17:42, David Leangen a écrit :
> 
> Hi there,
> 
> I think that my thoughts about components seem to be quite aligned with those of Eugen, so I won’t really repeat anything here. I agree with pretty much everything he writes about the advantages of having clean components. Perhaps the only thing I would point out is that even with clean APIs, you can still “mix up” the implementations. There is nothing prohibiting using a single “super protocol implement” that implements all of the protocols, even if the APIs are very cleanly separated.
> 
> I will only repeat that to have good componentization, independent (i.e. non-coupled) and very clear APIs are a necessary prerequisite. So to really get the most of Guice, I think there is some work to be done.
> 
> 
> Regarding the hexagonal architecture, I am not quite sure what to say. It sounds good on paper, but IMO any good architecture should help with communication. For the life of me I can’t seem to find my way around even knowing that it is based on this architecture. I don’t find any hints in the code. Can you direct me?
> 
> 
> However….
> 
> 
> Maybe given the current state of James we could just take a completely different approach. Instead of aiming for clean componentization, which would be a huge undertaking, and in any case the specs seem very “dirty” to me, and since James is already “working” maybe we could just use the “black box” approach. It is a closed system, so it doesn’t need APIs for its components.
> 
> The “interface” is simply a clean input and output function for emails.
> 
> Come to think of it, maybe that’s what the hexagonal architecture ends up doing??
> 
> 
> Cheers,
> =David
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

Hi there,

I think that my thoughts about components seem to be quite aligned with those of Eugen, so I won’t really repeat anything here. I agree with pretty much everything he writes about the advantages of having clean components. Perhaps the only thing I would point out is that even with clean APIs, you can still “mix up” the implementations. There is nothing prohibiting using a single “super protocol implement” that implements all of the protocols, even if the APIs are very cleanly separated.

I will only repeat that to have good componentization, independent (i.e. non-coupled) and very clear APIs are a necessary prerequisite. So to really get the most of Guice, I think there is some work to be done.


Regarding the hexagonal architecture, I am not quite sure what to say. It sounds good on paper, but IMO any good architecture should help with communication. For the life of me I can’t seem to find my way around even knowing that it is based on this architecture. I don’t find any hints in the code. Can you direct me?


However….


Maybe given the current state of James we could just take a completely different approach. Instead of aiming for clean componentization, which would be a huge undertaking, and in any case the specs seem very “dirty” to me, and since James is already “working” maybe we could just use the “black box” approach. It is a closed system, so it doesn’t need APIs for its components.

The “interface” is simply a clean input and output function for emails.

Come to think of it, maybe that’s what the hexagonal architecture ends up doing??


Cheers,
=David

Re: James SMTP Model

[Eugen Stan <eu...@netdava.com>]

Hi Benoit,

La 05.07.2020 07:32, Tellier Benoit a scris:
>
> Le 02/07/2020 à 23:03, Eugen Stan a écrit :
>> Hello David,
>>
>> [...]
>>
>> Agian, I do think protocols should be independent since they have
>> different rules and share only some technical aspects.
> +1
>
> Actually IMAP is independent.
>
> LMTP is based on SMTP implementation.
>
> A current limitation of LMTP is that it do not apply the mailet
> processing pipeline.
LMTP is based on SMTP so no surprise there.
>
> POP3, SMTP and ManageSieve relies on the common protocols-api.
>
>> The 'business' rules should trump over the technical part, but the code
>> is quite old on that part and it works so I guess no-one put the work to
>> change it much.
> +1 SMTP, LMTP and POP3 works globally well.
>
>> I believe each protocol is independent and should build to an
>> independent server (SMTP, IMAP, POP3, JMAP) .
> That statement is more questionable IMO.
>
> An IMAP server without an SMTP server will not be able to receive mails.
>
> While I agree on differentiating servers on the role they play (Mail
> Exchange, Mail Transfer, Mail Delivery, etc...) I believe fully
> splitting protocols into independent servers is not desirable.
>
> Today JAMES implementations makes sens to implement a Mail Transfer
> agent, and mail processing (achievable with s/jpa-smtp/Mail Processing
> server/) and for Mail Delivery (all other flavors). I believe we are not
> mature enough for mail exchange.
>
> Maybe the intended role in the mail architecture could be added to the
> Antora documentation?

I believe you make the assumption that people use and want to use the
full plethora of protocols we have in James.

Sort of "All or nothing" approach.

Why do you think it is so?

I would argue that we should make little assumptions about how people
deploy their infrastructure.

Maybe I have an existing infrastructure that I know and trust for SMTP
and I just want to use the IMAP server from James.

Examples:

1. I use Amazon Simple Email Service (or Sendgrid or Mailgun) for
sending emails and I just want to be able to customize the IMAP/JMAP
experience and control how I store emails.

2. I could use James SMTP as a Gateway for my existing IMAP / Webmail setup.

3. I could use James SMTP as a Relay for my marketing SaaS platform.


But that is not the main argument for having them separate.

The main argument IMO is that the protocols are quite complex and having
the ability to work on just one protocol at a time has significant
advantages:

* Lower cognitive effort - I only need to care about this specific
protocol and it's implementation details, not the other protocols.

  This is one huge benefit.

* Faster development feedback cycle - If I work on IMAP or SMTP I should
get away with running the tests only for that part.

* Less risk - If I work and break SMTP - the other parts should work.
With the current project structure - it can be hard to say.

* Easier to adopt by new people - If people care only about SMTP, LMTP
or IMAP - they can contribute only to that. The entry barrier will be
smoother than what we have now.

* Easier to compose and adapt for new and interesting projects - If we
package them independently and together in a single product (Advanced
Server) we have a showcase of how you can do that. It will be clear that
they are meant to be used in both ways.


Having them as separate projects may have the downside of integrating
them together as integrating them requires some form of glue.

The good side is that we already pay for this integration effort now :).

We will have to do the effort of splitting them apart while also keeping
the full version.

This is doable and while it will take time it's not a hard problem IMO.


Our way of using James is not the only way.


>> [...]
>>
>> I don't believe the code is super clean but it does work and is quite
>> efficient and fast.
> Nice explanation!
>
> +1 that's a foundation we may not want to challenge too much.

Challange all foundations!

(one at a time, when it's time comes :D )

>> NOTE: Netty should be upgraded to the latest version, I can do that once
>> we are done with Gradle.
> +1 <3
>
> Likely a huge effort though...
You questioning my determination ?! (joke)

In time I will change the universe :D .


-- 
Eugen Stan
+40720 898 747 / netdava.com

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Le 02/07/2020 à 23:03, Eugen Stan a écrit :
> Hello David,
>
> [...]
> 
> Agian, I do think protocols should be independent since they have
> different rules and share only some technical aspects.

+1

Actually IMAP is independent.

LMTP is based on SMTP implementation.

A current limitation of LMTP is that it do not apply the mailet
processing pipeline.

POP3, SMTP and ManageSieve relies on the common protocols-api.

> 
> The 'business' rules should trump over the technical part, but the code
> is quite old on that part and it works so I guess no-one put the work to
> change it much.

+1 SMTP, LMTP and POP3 works globally well.

> 
> I believe each protocol is independent and should build to an
> independent server (SMTP, IMAP, POP3, JMAP) .

That statement is more questionable IMO.

An IMAP server without an SMTP server will not be able to receive mails.

While I agree on differentiating servers on the role they play (Mail
Exchange, Mail Transfer, Mail Delivery, etc...) I believe fully
splitting protocols into independent servers is not desirable.

Today JAMES implementations makes sens to implement a Mail Transfer
agent, and mail processing (achievable with s/jpa-smtp/Mail Processing
server/) and for Mail Delivery (all other flavors). I believe we are not
mature enough for mail exchange.

Maybe the intended role in the mail architecture could be added to the
Antora documentation?

> 
> [...]
> 
> I don't believe the code is super clean but it does work and is quite
> efficient and fast.

Nice explanation!

+1 that's a foundation we may not want to challenge too much.

> 
> NOTE: Netty should be upgraded to the latest version, I can do that once
> we are done with Gradle.

+1 <3

Likely a huge effort though...

> [...]

Regards,

Benoit

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

> Since I am pretty much stuck with the documentation efforts (as I have mentioned I need more input from the community if I am to move forward), what I think I’ll try to do next is write an “independent” API for SMTP. […]

I spent a few hours on this today, but already I think I am going to give up. 😬

RFC5321 is (1) quite complex, especially because it has a lot of options and references other specs, and (2) really poorly written IMO, which makes interpreting it much more difficult than something I am willing to invest my energies into.

I’ll try poking around in other places in the hope that members will reply to my various outstanding questions.

In the meantime, my respect for James has grown. Although there are issues with readability and componentization, the fact that it can implement this kind of specification at all is really quite a feat.


Personally, I think that we should aspire higher. James should provide a kind of “reference” API for SMTP/IMAP etc in Java. As far as I can tell, James is the de facto Java API. The quality of the APIs ought to match this role. But then again, now that I’ve tried to make sense of the spec in the form of code and quickly gave up, maybe it’s just a pipe dream and the current API is the best that’s ever gonna happen.

Cheers,
=David

Re: James SMTP Model

[Tellier Benoit <bt...@apache.org>]

Hello David,

Le 03/07/2020 à 04:30, David Leangen a écrit :
> 
> [...]
>
>> I don't believe the code is super clean but it does work and is quite
>> efficient and fast.
> 
> Yes, that is important, but again, this comment about being “efficient and fast” seems perfectly valid for an implementation, but not really for an API. The API should be more about expressing the model and drawing clear component boundaries. (It should not preclude a good implementation, of course, but should usually not force a particular implementation either.)
> 
>> I agree with the coupling and it is going to take some effort to go
>> through all of the modules and clean them up.
> 
> Since I am pretty much stuck with the documentation efforts (as I have mentioned I need more input from the community if I am to move forward), what I think I’ll try to do next is write an “independent” API for SMTP. My hope is that this will:

From what I recall, business class exposed by the API are pretty well
defined: it exposes some hooks one could implement.

The big issue here is more that the protocols-smtp project does mix
together the "port" (as in hexagonal architecture) and the
implementation / wiring of it.

I believe, from a documentation perspective much can be achieved by
extracting a 'protocols-smtp-api' exposing the classes I can implement
as a protocols-smtp user, and having the actual smtp protocol
implementation in a separated project.

I believe Gradle will also make this "API extraction" easier.

As part of your documentation effort, I can provide a listing of such
classes; while waiting for the module extraction.

> 
>  * Provide an example as to how to write a “clean” and independent API
>  * Help with documentation (because in a way, the API **is** the documentation because it should reflect the model very well)


Here I have several concerns:

 - We should document what exist, and not a possible desirable future
state that may not be reached.

I believe if some reasonably easy moves could be done to ease the
documentation process, we should give it a go.

(The above proposal tries to achieve this)

>  * Continue to use the “existing” code that “does work and is quite efficient and fast” to implement the new API
> > In the worst case we can throw it away, in the best case it will
provide an example of how to move forward with better componentization
of James.

Here I'm concerned. Not that throwing code and rewritting things is a
bad idea, but it needs to be seen as an investment.

The question I would ask myself is "what do we gain from it?" "what are
the costs?" and "Is it the right moment to do that investment".

Now, protocolHandlers is a widely used API to extend James behavior and
we should pay special care about API stability to prevent our users to
rewritte their extensions.

Regards,

Benoit




---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

Hi Eugen,

> Quite a long email :) .

Hehehe, yes. Good thing I stopped when I did. Glad it got some dialogue going. I very much appreciate your reply.

Just a few comments inline.


> I believe the protocols share the protocols-api beacuse they are
> all/mostly all line based protocols.
> 
> Like you, I do think this is not a good solution, we should use
> composition instead of inheritance.

Ok, glad to know I’m not alone.


> I believe each protocol is independent and should build to an
> independent server (SMTP, IMAP, POP3, JMAP) .

I’m glad you think so, too.



> We could compose these independent servers into a single running process
> since they share the same language and platform (Java).

From a “beginner’s” perspective (which I still am), that makes a whole lotta sense, and corresponds well with how I think a system ought to be built.



> I do believe some duplication is better than coupling (this also applies
> to dependency management).

Again, glad to know I am not alone.



> Regarding ProtocolHandlerChain, the API is driven by how Netty
> implements protocols so please read a bit about that
> https://netty.io/wiki/user-guide-for-4.x.html#wiki-h3-4 .

Let me just repeat what you wrote: “the API is driven by how Netty implements protocols”.

That is just craziness to me. An API should never be driven by an implementation, pretty much by definition. It is always the implementation that should, well, implement the API. I think that writing the API because Netty works a certain way is completely backwards.

Now, if you are saying that the API is “inspired by” Netty, then perhaps I could buy that. However, that seems to be really limiting the API. It should be possible to have different implementations of an API, but by including lower-level concepts it cuts off a lot of potential solutions.

The API should be as stable as possible, and still valid even if somebody finds a better way of implementing it. All other components should depend solely on the API (not the implementations), and the runtime system is built by composing the different implementations.

From the sounds of it, that is what you want to move towards, so it sounds like we’re on the same page.

Using Netty in the implementation because it is fast (or whatever) makes perfect sense, so nothing against Netty or adapting to how Netty works, but I think this again mixes the roles of API and implementation.



> I don't believe the code is super clean but it does work and is quite
> efficient and fast.

Yes, that is important, but again, this comment about being “efficient and fast” seems perfectly valid for an implementation, but not really for an API. The API should be more about expressing the model and drawing clear component boundaries. (It should not preclude a good implementation, of course, but should usually not force a particular implementation either.)



> I agree with the coupling and it is going to take some effort to go
> through all of the modules and clean them up.

Since I am pretty much stuck with the documentation efforts (as I have mentioned I need more input from the community if I am to move forward), what I think I’ll try to do next is write an “independent” API for SMTP. My hope is that this will:

 * Provide an example as to how to write a “clean” and independent API
 * Help with documentation (because in a way, the API **is** the documentation because it should reflect the model very well)
 * Continue to use the “existing” code that “does work and is quite efficient and fast” to implement the new API

In the worst case we can throw it away, in the best case it will provide an example of how to move forward with better componentization of James.


Cheers,
=David

Re: James SMTP Model

[Eugen Stan <eu...@netdava.com>]

Hello David,

Quite a long email :) .

I'll try to come up with some answers.

From what I recall, the protocols where implemented by Norman Maurer
mostly.

However you can't see that from the git history since it has been
removed with the migration from SVN -> GIT.

I do believe that is a big legal issue since code committers retain
their copyright and with the migration, we removed any link to them -
but it's another discussion. 

See my reply inline:


La 01.07.2020 12:27, David Leangen a scris:
> Thanks to those who have replied so far.
>
> I am going to try to ask better questions. What I am trying to figure out is how James models the SMTP concepts in the code. I am very particular in that I think it is extremely important to make the code as “readable” as possible. I wanted to hear the thoughts behind the current model.
>
>
> I will dive in and tell you what I see, and ask some more questions. :-)
>
> Please remember that I represent those who are new to James and who want to use it and understand it. I also want to be able to document it better, which means that I need to understand it better.
>
>
> I started my journey with investigating the SMTP protocol api and implementation in James. I am specifically looking at the protocols-smtp project. Here is the train of thoughts I had as I tried to navigate the code to figure out how James models smtp.
>
>
> The first place I naturally look is the package org.apache.james.protocols.smtp. The package seems to be well-named: it is appropriate and easy to understand.
>
> Within this package, there are sub-packages: core, dsn, and hook. Those are definitely not the first place I would look to understand the smtp api. Within the interfaces / classes in the “main” package (i.e. “org.apache.james.protocols.smtp”), I find:
>
>     AllButStartTlsLineBasedChannelHandler.java
>     AllButStartTlsLineChannelHandlerFactory.java
>     CommandInjectionDetectedException.java
>     MailAddress.java
>     MailAddressException.java
>     MailEnvelope.java
>     MailEnvelopeImpl.java
>     SMTPConfiguration.java
>     SMTPConfigurationImpl.java
>     SMTPProtocol.java
>     SMTPProtocolHandlerChain.java
>     SMTPResponse.java
>     SMTPRetCode.java
>     SMTPServerMBean.java
>     SMTPSession.java
>     SMTPSessionImpl.java
>     SMTPStartTlsResponse.java
>
> Within these, MailAddress and MailAddressException are deprecated, so I ignore them.
>
> My first reactions are:
>
>  * AllButStartTlsLineBasedChannelHandler* … wtf??? Ok, don’t want to get into that right now.
>  * Why are there impl classes in the main api package????
>  * Why is there an MBean class polluting the api???
>  * SMTPProtocol looks like the most sensical place to start, so…
>
> I look at SMTPProtocol. My thoughts are:
>
>  * Holy crap, this is a class, not an interface. What’s going on here??
>  * Ok, it implements org.apache.james.protocols.api.ProtocolImpl, which itself implements org.apache.james.protocols.api.Protocol
>      - Already this is confusing and there is a lot of coupling going on here. Arg.
>
> Ok, first let’s look at the Protocol API.
>
> Ok, this is an interface, so I am indeed in api territory. I feel a bit better. But why is the protocols-smtp api coupled to the protocols-api project? That is not clear. Is there really a benefit to this inheritance? Coupling apis is generally quite yucky. Is the idea that ALL protocols are the same?

I believe the protocols share the protocols-api beacuse they are
all/mostly all line based protocols.

Like you, I do think this is not a good solution, we should use
composition instead of inheritance.

> My thought is that even if there is overlap between the protocol apis, it is a bit dubious to create this kind of coupling just because it may save a few lines of code. It has a fairly high risk of creating a world of pain in terms of causing unnecessary coupling and more difficult dependency management later on.
>
>  —> I understand that there are “protocols” and that they each “do something”, but can somebody explain the decision to use this kind of inheritance to try to define all of the protocols as a single “thing", instead of having independent apis per protocol? Is there some benefit to this coupling that I am not seeing?
>
> Anyway, to continue…

Agian, I do think protocols should be independent since they have
different rules and share only some technical aspects.

The 'business' rules should trump over the technical part, but the code
is quite old on that part and it works so I guess no-one put the work to
change it much.

I believe each protocol is independent and should build to an
independent server (SMTP, IMAP, POP3, JMAP) .

We could compose these independent servers into a single running process
since they share the same language and platform (Java).

I do believe some duplication is better than coupling (this also applies
to dependency management).

> A Protocol has a ProtocolHandlerChain, a ProtocolConfiguration, and a ProtocolSession. Alrighty, let’s look at those.
>
> ProtocolHandlerChain - this is in the org.apache.james.protocols.api.handler package, which means that the “main” package depends on a sub-package. That seems backwards to me. Anyway, when I look at this interface, I kinda get a sense of what it does, but it is soooo abstract and there is so little documentation that really I can only guess as to what it is for.
>
> ProtocolConfiguration - this is also in the “main” package, which is fine. This is very abstract, but since it represents a “Configuration” I think that’s ok. However, the provided documentation is pretty much useless. It only repeats the name of the interface methods without giving any additional information as to why these methods exist in the first place.
>
> ProtocolSession - This opens up a whole new pandora’s box that I am not ready to investigate right now. I’ll have to try to bite this part of the elephant later.
>
Regarding ProtocolHandlerChain, the API is driven by how Netty
implements protocols so please read a bit about that
https://netty.io/wiki/user-guide-for-4.x.html#wiki-h3-4 .

In summary:

* Netty implements an interceptor chain.

* Each element of that chain is responsible for some processing and
builds a more rich data structure for the next element

* SMPT/ IMAP protocols are line based and each line is usually a command

* You implement a handler for each protocol command, and all the logic
is encapsulated there.

* Each protocol is essentially a chain of handlers, where each handler
will implement a protocol command.

I don't believe the code is super clean but it does work and is quite
efficient and fast.

NOTE: Netty should be upgraded to the latest version, I can do that once
we are done with Gradle.

> Ok, so back to… ummmm, where was I? I’m already lost, so I’ll have to start again at SMTPProtocol and try to find my way again.
>
>
> Well, maybe I’ve already said enough for now anyway. I’ll leave it here for now, await responses, and continue again later.
>
>
>
> My incomplete (and maybe unjustified??) conclusions so far:
>
>  * There is a lot of coupling that I don’t understand and that makes the api already more complex than I would like
>  * There are many very abstract concepts that make comprehension very difficult for me
>  * The api and implementation classes are all mixed together, which I find utterly confusing
>
> Not sure if this is helpful or not, but I think that the ability to comprehend the codebase is very important, especially in a large and complex system in which many people collaborate.
>
> Hopefully people can answer some of the questions I pose in my thoughts above.

I agree with the coupling and it is going to take some effort to go
through all of the modules and clean them up.

This can be done as part of the project re-organization that was
proposed in https://issues.apache.org/jira/browse/JAMES-3259

Also see https://issues.apache.org/jira/browse/JAMES-2510 .

I support those efforts and I would like to do that after we migrate to
Gradle which will help with all of the points above.

It has much better support for multi module projects.

-- 
Eugen Stan
+40720 898 747 / netdava.com

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

Thanks to those who have replied so far.

I am going to try to ask better questions. What I am trying to figure out is how James models the SMTP concepts in the code. I am very particular in that I think it is extremely important to make the code as “readable” as possible. I wanted to hear the thoughts behind the current model.


I will dive in and tell you what I see, and ask some more questions. :-)

Please remember that I represent those who are new to James and who want to use it and understand it. I also want to be able to document it better, which means that I need to understand it better.


I started my journey with investigating the SMTP protocol api and implementation in James. I am specifically looking at the protocols-smtp project. Here is the train of thoughts I had as I tried to navigate the code to figure out how James models smtp.


The first place I naturally look is the package org.apache.james.protocols.smtp. The package seems to be well-named: it is appropriate and easy to understand.

Within this package, there are sub-packages: core, dsn, and hook. Those are definitely not the first place I would look to understand the smtp api. Within the interfaces / classes in the “main” package (i.e. “org.apache.james.protocols.smtp”), I find:

    AllButStartTlsLineBasedChannelHandler.java
    AllButStartTlsLineChannelHandlerFactory.java
    CommandInjectionDetectedException.java
    MailAddress.java
    MailAddressException.java
    MailEnvelope.java
    MailEnvelopeImpl.java
    SMTPConfiguration.java
    SMTPConfigurationImpl.java
    SMTPProtocol.java
    SMTPProtocolHandlerChain.java
    SMTPResponse.java
    SMTPRetCode.java
    SMTPServerMBean.java
    SMTPSession.java
    SMTPSessionImpl.java
    SMTPStartTlsResponse.java

Within these, MailAddress and MailAddressException are deprecated, so I ignore them.

My first reactions are:

 * AllButStartTlsLineBasedChannelHandler* … wtf??? Ok, don’t want to get into that right now.
 * Why are there impl classes in the main api package????
 * Why is there an MBean class polluting the api???
 * SMTPProtocol looks like the most sensical place to start, so…

I look at SMTPProtocol. My thoughts are:

 * Holy crap, this is a class, not an interface. What’s going on here??
 * Ok, it implements org.apache.james.protocols.api.ProtocolImpl, which itself implements org.apache.james.protocols.api.Protocol
     - Already this is confusing and there is a lot of coupling going on here. Arg.

Ok, first let’s look at the Protocol API.

Ok, this is an interface, so I am indeed in api territory. I feel a bit better. But why is the protocols-smtp api coupled to the protocols-api project? That is not clear. Is there really a benefit to this inheritance? Coupling apis is generally quite yucky. Is the idea that ALL protocols are the same?

My thought is that even if there is overlap between the protocol apis, it is a bit dubious to create this kind of coupling just because it may save a few lines of code. It has a fairly high risk of creating a world of pain in terms of causing unnecessary coupling and more difficult dependency management later on.

 —> I understand that there are “protocols” and that they each “do something”, but can somebody explain the decision to use this kind of inheritance to try to define all of the protocols as a single “thing", instead of having independent apis per protocol? Is there some benefit to this coupling that I am not seeing?

Anyway, to continue…


A Protocol has a ProtocolHandlerChain, a ProtocolConfiguration, and a ProtocolSession. Alrighty, let’s look at those.

ProtocolHandlerChain - this is in the org.apache.james.protocols.api.handler package, which means that the “main” package depends on a sub-package. That seems backwards to me. Anyway, when I look at this interface, I kinda get a sense of what it does, but it is soooo abstract and there is so little documentation that really I can only guess as to what it is for.

ProtocolConfiguration - this is also in the “main” package, which is fine. This is very abstract, but since it represents a “Configuration” I think that’s ok. However, the provided documentation is pretty much useless. It only repeats the name of the interface methods without giving any additional information as to why these methods exist in the first place.

ProtocolSession - This opens up a whole new pandora’s box that I am not ready to investigate right now. I’ll have to try to bite this part of the elephant later.


Ok, so back to… ummmm, where was I? I’m already lost, so I’ll have to start again at SMTPProtocol and try to find my way again.


Well, maybe I’ve already said enough for now anyway. I’ll leave it here for now, await responses, and continue again later.



My incomplete (and maybe unjustified??) conclusions so far:

 * There is a lot of coupling that I don’t understand and that makes the api already more complex than I would like
 * There are many very abstract concepts that make comprehension very difficult for me
 * The api and implementation classes are all mixed together, which I find utterly confusing

Not sure if this is helpful or not, but I think that the ability to comprehend the codebase is very important, especially in a large and complex system in which many people collaborate.

Hopefully people can answer some of the questions I pose in my thoughts above.


Cheers,
=David

Re: James SMTP Model

[Raphaël Ouazana-Sustowski <ro...@apache.org>]

Hello,

It seems you missed at least one common use case. An external SMTP 
server should be allowed to contact your SMTP server to send you emails 
related to your domains. This is the remote delivery part, but for your 
own server.

For this use case you need a SMTP server that is listening on a 
normalized port, and without authentication.

Regards,

Raphaël.

Le 26/06/2020 à 10:29, Eugen Stan a écrit :
> Hello David,
>
> I'll give it a try, but my knowledge is not that extensive at the moment.
>
> SMTP is a protocol for for transfering email (as the name suggests :D )
> https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol .
>
> A short, probably not entirely correct summary follows, with references
> to Apache James codebase follows:
>
> Users connect to the SMTP and pass the email message that they want to send.
>
> The SMTP optionally validates user credentials (highly recommended,
> otherwise you are an Open Relay = spam gateway).
>
> The SMTP server will process the email to identify if it is local or
> remote email.
>
> A local email is email for a domain "known/managed" by the same admin as
> the SMTP server.
>
> This local email will be delivered to the users inbox directly by
> storing it in the mail storage.
>
> A remote email address - the domain part is not managed by the SMTP system.
>
> In this case the SMTP server will have to forward the email to another
> SMTP server.
>
> For remote delivery, this SMTP server will look for the MX record for
> the domain found in the email destination address.
>
> It will connect to that server and via SMTP protocol and send it the email.
>
> When doing remote delivery, the SMTP server usually performs DNS
> checking, DKIM validation, etc.
>
> The SMTP server uses spooling system - a processing queue (like the
> printer) for delivering emails since:
>
> - Email deliver fails and needs to be retried
>
> - Remote email delivery takes time and we need to store the email safely
> from the time we accept it from the mail user agent (MUA) until we
> deliver it to the mailbox or to another remote system.
>
>
> References to James artifacts:
>
> * Parsing of the SMTP protocol is done in protocols/smtp module .
>
> We leverage netty network library for protocol parsing.
>
> When we parse the protocol we transform string lines to SMTP
> commands/actions that we need to implement via code.
>
> This is the low level API independent of other Apache James modules.
>
> Inside James we implement and provide actions for each SMTP protocol
> command in the server/protocols/protocols-smtp module.
>
> This module hooks into James specific libraries and provides more meat
> to the API.
>
> **server/protocols/protocols-smtp** is normally what you need to get going.
>
> In the current James setup, we take the classes from
> server/protocols/protocols-smtp and use Guice ( a dependency injection /
> wiring framework) to instantiate the SMTP components and wire them
> together in a form that will work for us.
>
> The guice wiring is available in server/container/guice/protocols/smtp .
>
> The guice module is a thin wrapper that provides just the wiring and
> builds a reusable guice module.
>
> The above pattern is repeated all over James, so as you can see we do
> have some separation of API and implementation, but not as pure as you
> find in other places.
>
>
> Also SMTP needs a queue to spool email and we have several
> implementation of that: RabbitMQ, Apache ActiveMQ, File queue, etc
>
> See modules in server/queue .
>
> Also SMTP uses dns and we use the dns modules in James, see
> server/dns-service for that.
>
> Also SMTP needs to perform user authentication and that part is managed
> by server/data modules (See User API packages and classes in data-api ).
>
> All of the above modules follow the same guice pattern presented above.
>
>
> La 26.06.2020 08:45, David Leangen a scris:
>> Hello,
>>
>> As I mentioned during our call, I need your help so I can continue to move forward with the documentation.
>>
>> My first questions are about SMTP.
>>
>>
>> * How does James model SMTP?
>> * Where can I find SMTP related concepts in the code?
>> * What are the related modules?
>> * How would I build my own SMTP server?
>> * What other information is relevant to SMTP that I may want to include?
>>
>>
>> Thanks!
>> =David
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: James SMTP Model

[Eugen Stan <eu...@netdava.com>]

Hello David,

I'll give it a try, but my knowledge is not that extensive at the moment.

SMTP is a protocol for for transfering email (as the name suggests :D )
https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol .

A short, probably not entirely correct summary follows, with references
to Apache James codebase follows:

Users connect to the SMTP and pass the email message that they want to send.

The SMTP optionally validates user credentials (highly recommended,
otherwise you are an Open Relay = spam gateway).

The SMTP server will process the email to identify if it is local or
remote email.

A local email is email for a domain "known/managed" by the same admin as
the SMTP server.

This local email will be delivered to the users inbox directly by
storing it in the mail storage.

A remote email address - the domain part is not managed by the SMTP system.

In this case the SMTP server will have to forward the email to another
SMTP server.

For remote delivery, this SMTP server will look for the MX record for
the domain found in the email destination address.

It will connect to that server and via SMTP protocol and send it the email.

When doing remote delivery, the SMTP server usually performs DNS
checking, DKIM validation, etc.

The SMTP server uses spooling system - a processing queue (like the
printer) for delivering emails since:

- Email deliver fails and needs to be retried

- Remote email delivery takes time and we need to store the email safely
from the time we accept it from the mail user agent (MUA) until we
deliver it to the mailbox or to another remote system.


References to James artifacts:

* Parsing of the SMTP protocol is done in protocols/smtp module .

We leverage netty network library for protocol parsing.

When we parse the protocol we transform string lines to SMTP
commands/actions that we need to implement via code.

This is the low level API independent of other Apache James modules.

Inside James we implement and provide actions for each SMTP protocol
command in the server/protocols/protocols-smtp module.

This module hooks into James specific libraries and provides more meat
to the API.

**server/protocols/protocols-smtp** is normally what you need to get going.

In the current James setup, we take the classes from
server/protocols/protocols-smtp and use Guice ( a dependency injection /
wiring framework) to instantiate the SMTP components and wire them
together in a form that will work for us.

The guice wiring is available in server/container/guice/protocols/smtp .

The guice module is a thin wrapper that provides just the wiring and
builds a reusable guice module.

The above pattern is repeated all over James, so as you can see we do
have some separation of API and implementation, but not as pure as you
find in other places.


Also SMTP needs a queue to spool email and we have several
implementation of that: RabbitMQ, Apache ActiveMQ, File queue, etc

See modules in server/queue .

Also SMTP uses dns and we use the dns modules in James, see
server/dns-service for that.

Also SMTP needs to perform user authentication and that part is managed
by server/data modules (See User API packages and classes in data-api ).

All of the above modules follow the same guice pattern presented above.


La 26.06.2020 08:45, David Leangen a scris:
> Hello,
>
> As I mentioned during our call, I need your help so I can continue to move forward with the documentation.
>
> My first questions are about SMTP.
>
>
> * How does James model SMTP?
> * Where can I find SMTP related concepts in the code?
> * What are the related modules?
> * How would I build my own SMTP server?
> * What other information is relevant to SMTP that I may want to include?
>
>
> Thanks!
> =David
>
>
-- 
Eugen Stan
+40720 898 747 / netdava.com

Re: James SMTP Model

[David Leangen <ap...@leangen.net>]

> My first questions are about SMTP.


After a bit of resisting, I finally bit the bullet and downloaded IntelliJ. I am now able to compile and navigate my way through the code in an IDE, so I should be able to make better progress.


> * How does James model SMTP?
> * Where can I find SMTP related concepts in the code?
> * What are the related modules?


I noticed there there is a protocols project, which has a protocols-smtp sub-project. However, I noticed that there is also a james-server-protocols-smtp project.

Why are there two smtp protocol projects?



By the way, just as a remark, I noticed that in both cases there is no separation of API / implementation. 😬

Is there any intention to fix that in order to make the system more modular?



> * How would I build my own SMTP server?
> * What other information is relevant to SMTP that I may want to include?

Still open questions.


Cheers,
=David