You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by David Leangen <ap...@leangen.net> on 2020/07/07 12:42:06 UTC
Queuing vs. spooling
Hi!
Please take a look at this image:
—> https://james.apache.org/images/james-smtp-relay.png
I have several questions. 😀
First, is it correct to say that in this image, the “SMTP Service” is an MTA? Furthermore, is it correct to say that it’s the “terminal” MTA?
How does the mail get forwarded from the SMTP server to the Mail Queue? Would that be via LMTP? Or something else?
How does the mail get transferred from the queue to the spooler? And by the way, what the heck is the difference between the queue and the spooler??
After that, the part about the Mailet Container I think makes sense to me, but everything up to the point is not clear at all, at least not to me.
Thanks as always for helping me understand!
Cheers,
=David
Re: Queuing vs. spooling
Posted by David Leangen <ap...@leangen.net>.
>>> Hope it helps.
>>
>> Yes, quite a lot!!
Still true.
> Yes, the client (either a MUA or a MTA) is the remote process talking
> SMTP to our SMTP server via a TCP connection.
Thanks, that clarifies. I misread your last email. Now everything you wrote makes sense.
So would it be fair to say generally that an “SMTP Relay” is composed of:
* SMTP service (for interfacing with the outside via SMTP protocol)
* A mail queue
* Optionally a spooler
* A processing chain
Also please confirm: An SMTP Relay is an MTA.
Also, can anybody else answer:
> Are there any other important “parts” that I should be aware of?
Cheers,
=David
Re: Queuing vs. spooling
Posted by Matthieu Baechler <ma...@apache.org>.
On Tue, 2020-07-07 at 22:58 +0900, David Leangen wrote:
> > Hope it helps.
>
> Yes, quite a lot!!
>
> A few clarifications, please. 😀
>
>
>
> > SMTP Service is talking TCP with the client. When it is asked to
> > deliver a message, it simply calls `enqueue` on the MailQueue.
>
> Can you be more precise about what you mean by “client”?
Yes, the client (either a MUA or a MTA) is the remote process talking
SMTP to our SMTP server via a TCP connection.
>
> > As everything happens in the Java process, you don't have a
> > protocol,
> > just a method call.
>
> By the way, is the message communicated via TCP or via a method call.
The SMTP server talks TCP with the outside world. Then it calls parts
of James with method calls (the SMTP server is living in the same
process as the other services James is running)
> Sorry, I am a bit confused about the two statements you made above.
>
> Once the mail is received by the mail queue, does all the rest of the
> process happen via method calls, too?
Yes
> > The spooler is the thing taking messages from the queue for
> > processing.
> > The MailQueue allows to decouple the reception from the handling.
> > A spooler usually is able to concurrently process several mails.
>
> Thanks.
>
> Are there any other important “parts” that I should be aware of?
>
I don't know.
-- Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by Eugen Stan <eu...@netdava.com>.
Great.
And as you said, it is important that whoever can answer some questions,
please find the time to do it.
You are helping yourself - the community grows, James grows, we all get
to benefit.
La 09.07.2020 05:05, Tellier Benoit a scris:
> Le 09/07/2020 à 07:42, David Leangen a écrit :
>> Hi Eugen,
>>
>> Thanks for your support.
>>
>>> GIven the discussion around the specific topics I think we are getting
>>> our documentation.
>>> @David: If you can do just that: ask questions and compile the answers
>>> it would be a huge win for us.
>> Thanks, that is indeed the intent since the beginning. The ability to move forward depends on how much everybody participates in answering my (often stupid) questions.
>>
>> I am approaching this from a newbie’s perspective, simply because I am one. As I mentioned in the beginning, (although less and less so now) I have the “advantage” of knowing very little about how James works, so it is easier for me to imagine what it’s like for other newbies trying to learn James, and what needs to be done to help make James more usable and thus more popular.
>>
>> It does require time and patience from everybody though. I hope the efforts will turn out to be worthwhile.
> It definitely worth it.
>
> +1 on the conclusion of this thread.
>
> Benoit
>> Thanks for all the help so far. Please keep it coming!!
>>
>>
>> Cheers,
>> =David
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
--
Eugen Stan
+40720 898 747 / netdava.com
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by Tellier Benoit <bt...@apache.org>.
Le 09/07/2020 à 07:42, David Leangen a écrit :
> Hi Eugen,
>
> Thanks for your support.
>
>> GIven the discussion around the specific topics I think we are getting
>> our documentation.
>> @David: If you can do just that: ask questions and compile the answers
>> it would be a huge win for us.
> Thanks, that is indeed the intent since the beginning. The ability to move forward depends on how much everybody participates in answering my (often stupid) questions.
>
> I am approaching this from a newbie’s perspective, simply because I am one. As I mentioned in the beginning, (although less and less so now) I have the “advantage” of knowing very little about how James works, so it is easier for me to imagine what it’s like for other newbies trying to learn James, and what needs to be done to help make James more usable and thus more popular.
>
> It does require time and patience from everybody though. I hope the efforts will turn out to be worthwhile.
It definitely worth it.
+1 on the conclusion of this thread.
Benoit
>
> Thanks for all the help so far. Please keep it coming!!
>
>
> Cheers,
> =David
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by David Leangen <ap...@leangen.net>.
Hi Eugen,
Thanks for your support.
> GIven the discussion around the specific topics I think we are getting
> our documentation.
> @David: If you can do just that: ask questions and compile the answers
> it would be a huge win for us.
Thanks, that is indeed the intent since the beginning. The ability to move forward depends on how much everybody participates in answering my (often stupid) questions.
I am approaching this from a newbie’s perspective, simply because I am one. As I mentioned in the beginning, (although less and less so now) I have the “advantage” of knowing very little about how James works, so it is easier for me to imagine what it’s like for other newbies trying to learn James, and what needs to be done to help make James more usable and thus more popular.
It does require time and patience from everybody though. I hope the efforts will turn out to be worthwhile.
Thanks for all the help so far. Please keep it coming!!
Cheers,
=David
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by Eugen Stan <eu...@netdava.com>.
Hi.
I don't have something to add to this thread specifically, but I would
like to point out that:
GIven the discussion around the specific topics I think we are getting
our documentation.
All we need to do now is copy paste it, curate it and edited in asciidoc
format.
I'll even put in the links to he mailing list archive / thread as
reference.
@David: If you can do just that: ask questions and compile the answers
it would be a huge win for us.
You don't even have to be very creative about it: it's good material.
Once I'm done with the automation I would like for us to 'get together'
in a (virtual) party and migrate the old website and get the new one out
the door.
Also, I'll try to find some time in the next days to work together on
some specific tasks
I believe it will be better than alone :).
Regards,
La 08.07.2020 11:49, Tellier Benoit a scris:
> Le 08/07/2020 à 15:33, David Leangen a écrit :
>> Thanks.
>>
>> I love this James adventure. It is not boring. Every time I scratch the surface, a new concept pops out. 😀
>>
>> Ok, so digging in…
> :-)
>>> Also as Matthieu said, RemoteDelivery is a side effect of the current Processing chain.
>> I looked at org.apache.james.transport.mailets.RemoteDelivery, with the assumption that it is the correct Mailet.
> Correct
>> First, the javadoc says:
>>> The RemoteDelivery mailet delivers messages to a remote SMTP server able to deliver or forward messages to their final destination.
>> That is consistent with what you wrote. So I looked at the code to try to get a sense of *how* it does this.
> Ouch. Complex and old code relying on javax.mail is waiting you ^^
>> What I could gather is that the magic is actually done with org.apache.james.queue.api.MailQueue. All RemoteDelivery does is enqueue the Mail, then it’s done.
> It do start DeliveryRunnable that consume the outgoing queue. Relaying
> happens there.
>> So the queue-api seems now quite important.
> I confirm it is.
>> I also gather that it is the same queue that accepts mails from the incoming SMTP Service in the figure. However, by looking at the api code and the sparse javadocs that come with it, I am getting few clues as to what it does or how it works.
>>
>> I can only guess that the actual heavy lifting is done with an implementation that gets wired with Guice.
> It is guice agnostic. Look inside RemoteDelivery::init...
>> I will investigate further, but in the meantime, can somebody please point me in the right direction to help speed up my journey?
>>
> Did I just do that or you need more information?
>
> Cheers,
>
> Benoit
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
--
Eugen Stan
+40720 898 747 / netdava.com
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by Tellier Benoit <bt...@apache.org>.
Le 08/07/2020 à 15:33, David Leangen a écrit :
> Thanks.
>
> I love this James adventure. It is not boring. Every time I scratch the surface, a new concept pops out. 😀
>
> Ok, so digging in…
:-)
>> Also as Matthieu said, RemoteDelivery is a side effect of the current Processing chain.
> I looked at org.apache.james.transport.mailets.RemoteDelivery, with the assumption that it is the correct Mailet.
Correct
> First, the javadoc says:
>> The RemoteDelivery mailet delivers messages to a remote SMTP server able to deliver or forward messages to their final destination.
> That is consistent with what you wrote. So I looked at the code to try to get a sense of *how* it does this.
Ouch. Complex and old code relying on javax.mail is waiting you ^^
> What I could gather is that the magic is actually done with org.apache.james.queue.api.MailQueue. All RemoteDelivery does is enqueue the Mail, then it’s done.
It do start DeliveryRunnable that consume the outgoing queue. Relaying
happens there.
> So the queue-api seems now quite important.
I confirm it is.
> I also gather that it is the same queue that accepts mails from the incoming SMTP Service in the figure. However, by looking at the api code and the sparse javadocs that come with it, I am getting few clues as to what it does or how it works.
>
> I can only guess that the actual heavy lifting is done with an implementation that gets wired with Guice.
It is guice agnostic. Look inside RemoteDelivery::init...
> I will investigate further, but in the meantime, can somebody please point me in the right direction to help speed up my journey?
>
Did I just do that or you need more information?
Cheers,
Benoit
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by David Leangen <ap...@leangen.net>.
Thanks.
I love this James adventure. It is not boring. Every time I scratch the surface, a new concept pops out. 😀
Ok, so digging in…
> Also as Matthieu said, RemoteDelivery is a side effect of the current Processing chain.
I looked at org.apache.james.transport.mailets.RemoteDelivery, with the assumption that it is the correct Mailet.
First, the javadoc says:
> The RemoteDelivery mailet delivers messages to a remote SMTP server able to deliver or forward messages to their final destination.
That is consistent with what you wrote. So I looked at the code to try to get a sense of *how* it does this. What I could gather is that the magic is actually done with org.apache.james.queue.api.MailQueue. All RemoteDelivery does is enqueue the Mail, then it’s done.
So the queue-api seems now quite important. I also gather that it is the same queue that accepts mails from the incoming SMTP Service in the figure. However, by looking at the api code and the sparse javadocs that come with it, I am getting few clues as to what it does or how it works.
I can only guess that the actual heavy lifting is done with an implementation that gets wired with Guice.
I will investigate further, but in the meantime, can somebody please point me in the right direction to help speed up my journey?
Thanks!
=David
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by Tellier Benoit <bt...@apache.org>.
Le 08/07/2020 à 14:39, David Leangen a écrit :
>> The only wrong thing about this picture is the SMTP Service before "Outgoing".
> Ok, thank you Matthieu.
>
>> As weird as it is, the delivery of messages to a remote server is done by a Mailet called RemoteDelivery and it's not handled by the SMTP Service.
>> As far as I know, a lot of people are actually forking this Mailet because they want specific behaviors for delivery so I think this design makes sense.
> Ok sure, but doesn’t that Mailet have to “speak” SMTP?
No they don't.
They encapsulate what a "Mail" (a Mime message with its transport
context) is and transform it.
They can for instance be used after a JMAP submission servers (as users
can send mails via JMAP too)
Mailet are just a very simple Java interface. There is no hard
dependency to SMTP.
> I.e., doesn’t it effectively become an SMTP client, that opens a session with a remote SMTP server?
>
> Or am I again misunderstanding something fundamental about how SMTP works?
>
>
> Cheers,
> =David
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by David Leangen <ap...@leangen.net>.
> The only wrong thing about this picture is the SMTP Service before "Outgoing".
Ok, thank you Matthieu.
> As weird as it is, the delivery of messages to a remote server is done by a Mailet called RemoteDelivery and it's not handled by the SMTP Service.
> As far as I know, a lot of people are actually forking this Mailet because they want specific behaviors for delivery so I think this design makes sense.
Ok sure, but doesn’t that Mailet have to “speak” SMTP? I.e., doesn’t it effectively become an SMTP client, that opens a session with a remote SMTP server?
Or am I again misunderstanding something fundamental about how SMTP works?
Cheers,
=David
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by Matthieu Baechler <ma...@apache.org>.
On Wed, 2020-07-08 at 16:21 +0900, David Leangen wrote:
> Still on the topic of SMTP relay and the original image I posted at
> the beginning of this thread:
> —> https://james.apache.org/images/james-smtp-relay.png
>
>
> Would the attached (lousy) image be a reasonable representation of
> the general concept of an SMTP Relay?
>
> Still on the topic of SMTP relay and the original image I posted at
> the beginning of this thread:
>
> —> https://james.apache.org/images/james-smtp-relay.png
>
>
> Would the attached (lousy) image be a reasonable representation of
> the general concept of an SMTP Relay?
>
The only wrong thing about this picture is the SMTP Service before
"Outgoing".As weird as it is, the delivery of messages to a remote
server is done by a Mailet called RemoteDelivery and it's not handled
by the SMTP Service.As far as I know, a lot of people are actually
forking this Mailet because they want specific behaviors for delivery
so I think this design makes sense.
> -- Matthieu Baechler
Re: SMTP Relay (Was: Queuing vs. spooling)
Posted by Tellier Benoit <bt...@apache.org>.
Nice image!
- there is a queue before SMTP outgoing service (as you don't want to
handle SMTP synchronously anyway)
client -> SMTP (in, as a server) -> queue (spool) -> Processing chain ->
Queue (outgoing) -> RemoteDelivery runnable
Also as Matthieu said, RemoteDelivery is a side effect of the current
Processing chain.
Cheers,
Benoit
Le 08/07/2020 à 14:21, David Leangen a écrit :
>
> Still on the topic of SMTP relay and the original image I posted at
> the beginning of this thread:
>
> —> https://james.apache.org/images/james-smtp-relay.png
>
>
> Would the attached (lousy) image be a reasonable representation of the
> general concept of an SMTP Relay?
>
SMTP Relay (Was: Queuing vs. spooling)
Posted by David Leangen <ap...@leangen.net>.
Still on the topic of SMTP relay and the original image I posted at the beginning of this thread:
—> https://james.apache.org/images/james-smtp-relay.png
Would the attached (lousy) image be a reasonable representation of the general concept of an SMTP Relay?
Re: Queuing vs. spooling
Posted by Tellier Benoit <bt...@apache.org>.
Le 07/07/2020 à 20:58, David Leangen a écrit :
>> Hope it helps.
> Yes, quite a lot!!
>
> A few clarifications, please. 😀
>
>> SMTP Service is talking TCP with the client. When it is asked to
>> deliver a message, it simply calls `enqueue` on the MailQueue.
> Can you be more precise about what you mean by “client”?
By SMTP client Matthieu means either a Mail User Agent (eg Thunderbird)
or a remote mail server transferring emails to our server, thus sending
SMTP client commands.
>> As everything happens in the Java process, you don't have a protocol,
>> just a method call.
> By the way, is the message communicated via TCP or via a method call. Sorry, I am a bit confused about the two statements you made above.
>
> Once the mail is received by the mail queue, does all the rest of the process happen via method calls, too?
Via method calls. protocols/protocols-smtp library is agnostic about it,
but server/protocols/protocols-smtp does the link between the smtp
library and other James components (was it your question?)
>> The spooler is the thing taking messages from the queue for processing.
>> The MailQueue allows to decouple the reception from the handling.
>> A spooler usually is able to concurrently process several mails.
>
> Thanks.
>
> Are there any other important “parts” that I should be aware of?
Regarding what?
>
>
> Cheers,
> =David
>
Cheers,
Benoit
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Queuing vs. spooling
Posted by David Leangen <ap...@leangen.net>.
> Hope it helps.
Yes, quite a lot!!
A few clarifications, please. 😀
> SMTP Service is talking TCP with the client. When it is asked to
> deliver a message, it simply calls `enqueue` on the MailQueue.
Can you be more precise about what you mean by “client”?
> As everything happens in the Java process, you don't have a protocol,
> just a method call.
By the way, is the message communicated via TCP or via a method call. Sorry, I am a bit confused about the two statements you made above.
Once the mail is received by the mail queue, does all the rest of the process happen via method calls, too?
> The spooler is the thing taking messages from the queue for processing.
> The MailQueue allows to decouple the reception from the handling.
> A spooler usually is able to concurrently process several mails.
Thanks.
Are there any other important “parts” that I should be aware of?
Cheers,
=David
Re: Queuing vs. spooling
Posted by Matthieu Baechler <ma...@apache.org>.
Hi
On Tue, 2020-07-07 at 21:42 +0900, David Leangen wrote:
> Hi!
>
> Please take a look at this image:
>
> —> https://james.apache.org/images/james-smtp-relay.png
>
>
> I have several questions. 😀
>
>
> First, is it correct to say that in this image, the “SMTP Service” is
> an MTA? Furthermore, is it correct to say that it’s the “terminal”
> MTA?
The SMTP service on this schema is just the server you talk to on port
25. The full picture is an MTA. And in this case it's just a relay with
some rules.
> How does the mail get forwarded from the SMTP server to the Mail
> Queue? Would that be via LMTP? Or something else?
SMTP Service is talking TCP with the client. When it is asked to
deliver a message, it simply calls `enqueue` on the MailQueue.
As everything happens in the Java process, you don't have a protocol,
just a method call.
>
> How does the mail get transferred from the queue to the spooler?
The spooler basically pull messages from the Queue and then handles
them.
> And by the way, what the heck is the difference between the queue
> and the spooler??
The spooler is the thing taking messages from the queue for processing.
The MailQueue allows to decouple the reception from the handling.
A spooler usually is able to concurrently process several mails.
> After that, the part about the Mailet Container I think makes sense
> to me, but everything up to the point is not clear at all, at least
> not to me.
>
>
> Thanks as always for helping me understand!
>
Hope it helps.
Cheers,
-- Matthieu Baechler
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org