Moving mails around (Was: MailRepository and MailboxManager)

[Stefano Bagnara <ap...@bago.org>]

2010/1/22 Robert Burrell Donkin <ro...@gmail.com>:
> On Fri, Jan 22, 2010 at 8:52 AM, Stefano Bagnara <ap...@bago.org> wrote:
>> 2010/1/21 Robert Burrell Donkin <ro...@gmail.com>:
>>> i think that using sieve to deliver IMAP mail is wrong. i think that
>>> the script should set attributes on the mail (for example, set a
>>> attribute for folder) and with independent delivery.
>>
>> 5 years ago I was proposing
>> (http://markmail.org/message/xkcttgyqmfwpieew) to use a "destination"
>> concept for moving mails around. I think we could use special email
>> addresses (and maybe "forward paths", with reference to the "old"
>> source routing stuff) so to still keep working with mail envelopes
>> (even using a distributed email processor, as multiple james servers)
>> or alternatively using urls. The fact is that "state" is not enough to
>> deal with message routing so we need something more in the mailet api,
>> and we can do this 2 ways: A. add routing methods, B. add some more
>> expressive property than state.
>
> sounds very interesting. could you write up the design more fully?

Currently we have multiple ways we move emails around:

1) alter the "Mail.state" let us moving from one processor to another
(the most used one)
2) call MailetContext.sendMail/MailServer.sendMail to put a copy or
the male itself back in the spool (usually done in the
forward/autoreply/notify mailets.
3) lookup a repository from the store and put that message to the
repository (LocalDelivery, ToRepository, part of RemoteDelivery)
4) have a mailet with special code that transmit the message content
using a custom way (like part of RemoteDelivery)

James (the spoolmanager) have complete control in case 1 (it is the
spoolmanager to really move the message) and partial control on 2 (the
spoolmanager knows this, but it doesn't know what is the relationship
between the mail being processed and the mail being sent via
sendMail). THe spool does not have control on what happens in 3 and 4.

Furthermore 3 is something we all like agreed we should fix (remove)
because this makes many mailets to be james specific and not generic
mailets. In past we had contrasting proposals: A. add repositories to
the mailet api, B. extending the state/sendMail behaviour so that
inbox delivery is supported without looking up the Store/Repositoryes.

Then we have something that we would like to "add" to the current
functionalities:
1) being able to process messages based only on their envelope or only
on the message headers (in contrast to processing full
Mail+MimeMessage objects like we do now)
2) being able to track the flow of a message (so that the spoolmanager
can tell us if a message we sent it has been processed and if it is
processed what is the status: failed, relayed, delivered to remote
host, delivered locally, explanded to list of recipients... etc...).
This one is something that would allow us to really implement DSN, but
I'm not sure that DSN worth this complexity, so I simply share this
possible "requirement" but I'm not saying we should really try to fix
(implement) this.
3) We all agreed that we should stop moving around the whole
Mail+MimeMessage and instead split this so to move less data as
possible. From my understanding this means that an incoming message is
placed in a messagestore and its envelope(+maybe metadata) is moved
around. I'm not sure whether "inbox delivery" will eventually move the
message or if the message will be still referenced to the original
place we wrote it, but we all know that we should stop copying the
full message at every spooling choice.

So, this is the list of current usecases, I need more time to
elaborate possible improvements, so I just share this by now. Maybe
you like to add usecases, too.

Stefano

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Moving mails around (Was: MailRepository and MailboxManager)

[Norman Maurer <no...@googlemail.com>]

Ok some things we should really try todo (many thigns already said by Stefano):

1) I like the idea of only moving the "envelope" stuff around. Most
time we have no need to "copy,move" the whole message
2) Share the same "stored" message for all recipients (maybe store the
only the diff if the message change for some recipients)
3) write the message to disk (whatever implementation) and keep a copy
in memory if the message is not to big to save the I/O
4) Not use javamail where ever possible.

Bye,
Norman


2010/1/22 Robert Burrell Donkin <ro...@gmail.com>:
> On Fri, Jan 22, 2010 at 1:40 PM, Stefano Bagnara <ap...@bago.org> wrote:
>> 2010/1/22 Robert Burrell Donkin <ro...@gmail.com>:
>>> On Fri, Jan 22, 2010 at 8:52 AM, Stefano Bagnara <ap...@bago.org> wrote:
>>>> 2010/1/21 Robert Burrell Donkin <ro...@gmail.com>:
>>>>> i think that using sieve to deliver IMAP mail is wrong. i think that
>>>>> the script should set attributes on the mail (for example, set a
>>>>> attribute for folder) and with independent delivery.
>>>>
>>>> 5 years ago I was proposing
>>>> (http://markmail.org/message/xkcttgyqmfwpieew) to use a "destination"
>>>> concept for moving mails around. I think we could use special email
>>>> addresses (and maybe "forward paths", with reference to the "old"
>>>> source routing stuff) so to still keep working with mail envelopes
>>>> (even using a distributed email processor, as multiple james servers)
>>>> or alternatively using urls. The fact is that "state" is not enough to
>>>> deal with message routing so we need something more in the mailet api,
>>>> and we can do this 2 ways: A. add routing methods, B. add some more
>>>> expressive property than state.
>>>
>>> sounds very interesting. could you write up the design more fully?
>>
>> Currently we have multiple ways we move emails around:
>>
>> 1) alter the "Mail.state" let us moving from one processor to another
>> (the most used one)
>> 2) call MailetContext.sendMail/MailServer.sendMail to put a copy or
>> the male itself back in the spool (usually done in the
>> forward/autoreply/notify mailets.
>> 3) lookup a repository from the store and put that message to the
>> repository (LocalDelivery, ToRepository, part of RemoteDelivery)
>> 4) have a mailet with special code that transmit the message content
>> using a custom way (like part of RemoteDelivery)
>>
>> James (the spoolmanager) have complete control in case 1 (it is the
>> spoolmanager to really move the message) and partial control on 2 (the
>> spoolmanager knows this, but it doesn't know what is the relationship
>> between the mail being processed and the mail being sent via
>> sendMail). THe spool does not have control on what happens in 3 and 4.
>>
>> Furthermore 3 is something we all like agreed we should fix (remove)
>> because this makes many mailets to be james specific and not generic
>> mailets. In past we had contrasting proposals: A. add repositories to
>> the mailet api, B. extending the state/sendMail behaviour so that
>> inbox delivery is supported without looking up the Store/Repositoryes.
>>
>> Then we have something that we would like to "add" to the current
>> functionalities:
>> 1) being able to process messages based only on their envelope or only
>> on the message headers (in contrast to processing full
>> Mail+MimeMessage objects like we do now)
>> 2) being able to track the flow of a message (so that the spoolmanager
>> can tell us if a message we sent it has been processed and if it is
>> processed what is the status: failed, relayed, delivered to remote
>> host, delivered locally, explanded to list of recipients... etc...).
>> This one is something that would allow us to really implement DSN, but
>> I'm not sure that DSN worth this complexity, so I simply share this
>> possible "requirement" but I'm not saying we should really try to fix
>> (implement) this.
>> 3) We all agreed that we should stop moving around the whole
>> Mail+MimeMessage and instead split this so to move less data as
>> possible. From my understanding this means that an incoming message is
>> placed in a messagestore and its envelope(+maybe metadata) is moved
>> around. I'm not sure whether "inbox delivery" will eventually move the
>> message or if the message will be still referenced to the original
>> place we wrote it, but we all know that we should stop copying the
>> full message at every spooling choice.
>>
>> So, this is the list of current usecases, I need more time to
>> elaborate possible improvements, so I just share this by now. Maybe
>> you like to add usecases, too.
>
> i'll look forward to that :-)
>
> thanks
>
> - robert
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Moving mails around (Was: MailRepository and MailboxManager)

[Robert Burrell Donkin <ro...@gmail.com>]

On Fri, Jan 22, 2010 at 1:40 PM, Stefano Bagnara <ap...@bago.org> wrote:
> 2010/1/22 Robert Burrell Donkin <ro...@gmail.com>:
>> On Fri, Jan 22, 2010 at 8:52 AM, Stefano Bagnara <ap...@bago.org> wrote:
>>> 2010/1/21 Robert Burrell Donkin <ro...@gmail.com>:
>>>> i think that using sieve to deliver IMAP mail is wrong. i think that
>>>> the script should set attributes on the mail (for example, set a
>>>> attribute for folder) and with independent delivery.
>>>
>>> 5 years ago I was proposing
>>> (http://markmail.org/message/xkcttgyqmfwpieew) to use a "destination"
>>> concept for moving mails around. I think we could use special email
>>> addresses (and maybe "forward paths", with reference to the "old"
>>> source routing stuff) so to still keep working with mail envelopes
>>> (even using a distributed email processor, as multiple james servers)
>>> or alternatively using urls. The fact is that "state" is not enough to
>>> deal with message routing so we need something more in the mailet api,
>>> and we can do this 2 ways: A. add routing methods, B. add some more
>>> expressive property than state.
>>
>> sounds very interesting. could you write up the design more fully?
>
> Currently we have multiple ways we move emails around:
>
> 1) alter the "Mail.state" let us moving from one processor to another
> (the most used one)
> 2) call MailetContext.sendMail/MailServer.sendMail to put a copy or
> the male itself back in the spool (usually done in the
> forward/autoreply/notify mailets.
> 3) lookup a repository from the store and put that message to the
> repository (LocalDelivery, ToRepository, part of RemoteDelivery)
> 4) have a mailet with special code that transmit the message content
> using a custom way (like part of RemoteDelivery)
>
> James (the spoolmanager) have complete control in case 1 (it is the
> spoolmanager to really move the message) and partial control on 2 (the
> spoolmanager knows this, but it doesn't know what is the relationship
> between the mail being processed and the mail being sent via
> sendMail). THe spool does not have control on what happens in 3 and 4.
>
> Furthermore 3 is something we all like agreed we should fix (remove)
> because this makes many mailets to be james specific and not generic
> mailets. In past we had contrasting proposals: A. add repositories to
> the mailet api, B. extending the state/sendMail behaviour so that
> inbox delivery is supported without looking up the Store/Repositoryes.
>
> Then we have something that we would like to "add" to the current
> functionalities:
> 1) being able to process messages based only on their envelope or only
> on the message headers (in contrast to processing full
> Mail+MimeMessage objects like we do now)
> 2) being able to track the flow of a message (so that the spoolmanager
> can tell us if a message we sent it has been processed and if it is
> processed what is the status: failed, relayed, delivered to remote
> host, delivered locally, explanded to list of recipients... etc...).
> This one is something that would allow us to really implement DSN, but
> I'm not sure that DSN worth this complexity, so I simply share this
> possible "requirement" but I'm not saying we should really try to fix
> (implement) this.
> 3) We all agreed that we should stop moving around the whole
> Mail+MimeMessage and instead split this so to move less data as
> possible. From my understanding this means that an incoming message is
> placed in a messagestore and its envelope(+maybe metadata) is moved
> around. I'm not sure whether "inbox delivery" will eventually move the
> message or if the message will be still referenced to the original
> place we wrote it, but we all know that we should stop copying the
> full message at every spooling choice.
>
> So, this is the list of current usecases, I need more time to
> elaborate possible improvements, so I just share this by now. Maybe
> you like to add usecases, too.

i'll look forward to that :-)

thanks

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org