MessageRepository and key, msn, uid, Flags (Was: IMAP development)

[Stefano Bagnara <ap...@bago.org>]

Joachim Draeger wrote:
>> Is EventTriggerMailbox so important for basic proof of concept usage?
>> Is the whole "column selection" stuff (MessageResult) / 
>> messageSetTypes a need for the basic repository interface?
> 
> Imap requires to deliver events about added/deleted and changed flags to 
> the client.
> It would be possible to fake this events by just acknowledging actions 
> from the client by an appropriate event (this is what the client relies 
> on). But then you would have to manage workaround code.

Ok, I know I'm ignorant about IMAP. Sorry for so much questions!

>>> I'm quite curious what you think about GeneralMessageSet and 
>>> MessageResult.
>>> I started a JDBC prototype and found it quite handy.
>>> Well, the only thing from M1 that could be postponed is the Javamail 
>>> proof-of-concept.
>>
>> I don't like GeneralMessageSet becase:
>> 1) Why do we need 3 types of keys (uid/msn/key) in a generic message 
>> repository?
> 
> uid is required by imap. It's not trivial to implement uid capabilities 
> for all repository backends. You can translate uid to key (by just using 
> a string representation) but not vise versa.
> So "clients" that don't need uid can access everything safely by key or 
> msn (message number).

Sorry but I'm slow to understand this.

KEY => A unique string that identify a single message in the repository: 
persistent over sessions and restart, right?

UID => A unique number that identify a single message in the repository: 
persistent over sessions and restart as long as UIDVALIDITY does not 
change (can UIDVALIDITY be changed in the middle of a session?)

MSN => Are temporary numbers having meanings only inside a given session 
and uniquely associated to an underlying message (From my understanding 
we cannot use UIDs as MSN because MSN must start with 1 and be a 
complete sequence, is this right?).

So MSN is the message number that we also use when deleting messages by 
message number in pop3. What about POP3 UIDL ? POP3 require persistent 
keys: can we use UID or do we need KEYs for this?

I would like to understand the differences between UID and KEY and why 
we need both. Can't we have one of them to implement the other?

>> 2) Imo the name is misleading: It is not a messageSet but a 
>> SearchCriteria.
> 
> No, I don't think so. MSN to MSN, UID to UID, one key, is always an 
> exact definition of one or a range of messages.

Well, the fact that in some circumstances the search identify a specific 
message or a specific set is not a problem: in fact when you run a 
SELECT query and put a WHERE over the primary key you obtain a single 
result, but it is anyway a QUERY to search records.

I prefer SearchCriteria, but this is not important at all: when we'll 
have finished any other more important issue we'll discuss names ;-).

>> About MessageResult:
>> 1) I think we should not reference Mail objects here, but only 
>> MimeMessage objects (or even simple streams)
> 
> The idea is, when fetching from a spool you can get the Mail object. (if 
> you have requested it, if not possible, because it is not a spool, the 
> request for a Mail object had thrown an exception)

Imo we should remove this concept of SPOOL from the message repository. 
If one will ever want to implement a spool repository based on a 
messagereposirory it will have to store the Mail informations in a 
messagerepository way (as Flags, or as custom MimeHeaders). 
MessageRepository should not have dependencies on the Mail object.

> And yes I just thought about including a stream as a result.
> JavaMail for example is designed very object orientated, which is nice 
> for the client application.
> For the backend it is horrible: You have to provide a light-weight 
> MimeMessage, that lazy-loads headers, flags, content, uid etc on demand.

We already have MimeMessageWrapper and we already delay the loading of 
the message body by using a MimeMessageSource. The remaining data should 
not be so much stuff to not be loaded everytime: isn't it?

> You always have to check things like: is the mailbox/message still there?
> Maybe you have to reopen resources do hashing/cashing/parsing each 
> request...
> You can't use a efficient SQL JOIN to fetch everything requested at once.
 >
> I noticed that you know mostly exactly what you want, e.g.: flags, 
> MimeMessage, Uid
> So the backend *can* optimize this request to fetch everything at once.
> And if you just want stream and flags, it should provide you with stream 
> and flags and not an all-purpose pseudo-light-weight MimeMessage 
> monster. ;-)
> And not require you to do a request for every single item.

Can't we load everything but the mimemessage and keep the lazy loading 
(MimeMessageSource) for this one?
Is it so much work to load flags and uid for each message? Can't we do 
this as a first step, and optimize things later?

>> 2) Why Key/Flags/Msn/Uid/InternalDate are present in this interface 
>> and, as an example, the From or the Subject is not there? Is this 
>> related to what the IMAP protocol needs?
> 
> Oh yes, I thought about requesting headers without MimeMessage monster, 
> which maybe prepares resources to fetch a body stream...  :-)
> And no: key is what James needs, msn is what pop3+imap needs
> And Yes, the rest is needed by IMAP.

Remember that we already have a MimeMessageWrapper: we could work on it 
to know what is already loaded and what will need to load the headers or 
the body. We already do something similar (We disabled the header 
lazyness in 2.3 because of a bug that I have to investigate yet).

>> 3) It seems TOO much IMAP oriented to not have Imap in the name: I 
>> don't think that (as an example) an NNTPMessageRepository would need 
>> Msn/Uid/Key/Flags and maybe it would need something else
> 
> NNTP uses article numbers. I just flew over the rfc and it talks about a 
> first and a last article number in the newsgroups so it is more uid like.
> I guess NNTP would use message-ids as keys.
> And when you access an NNTP repository through pop3/imap or custom code 
> in a mailet you could use msns.

I still don't have this msn/uid/key stuff clear in my mind...

>> That said I'm not against making an IMAP specific repository and I 
>> always said that we should have a clear goal (Imap support) and create 
>> a repository based on the specific needs of Imap (or better the 
>> specific uses the most common Imap clients make of the Imap server).
> 
> msn/uid/flags, hierarchy like structure.

Ok, we need hierarchy like also in the core james, so we should find a 
generic way to reference message repositories subtrees and to handle 
virtual trees.

Tell me something more about flags: can we simplify them as key=>value 
as string=>string pairs? Or what else?

<snipped some mailbox considerations: I'll reply to them later>

>> SpoolRepository and MessageRepositories will not be related any more. 
>> I don't think that we'll ever need/want to write a 
>> SpoolRepositoryAsMessageRepositoryWrapper.
> 
> I'm not up to date on the latest spool discussion. But for me a spool is 
> a MessageRepository with additional attributes and additional query 
> possibilities.
> So if I would have written a great MessageRepository implementation I 
> could extend it to be able to work as an SpoolRepository.
> But this is a bit out of scope for now. :-)

We'll try to refactor SpoolRepositories so that they *uses* a 
MessageRepository for the MimeMessage but they only contains and spool 
the Envelope part (the Mail object excluding the message): this would 
mean that you can move a Mail object from a spool to another without 
moving the related message (if both spool repositories uses the same 
message repository).

>>> To have something more concrete I just started to implement a Apache 
>>> Torque/JDBC  based backend. It already helped me to find out a few 
>>> weak points of the design.
>>
>> Great news!
>> Good choice with Torque as it is license compatible with James!
> 
> BTW I'm a bit disappointed. I tested Torque in the past and used it's 
> PHP clone Propel in a bigger project. And now I had to find out that 
> Propel has feature-overtook Torque. :-)
> But I'm still convinced that even if there are not so many tables, it's 
> more agile not to have to write so much SQL and JDBC specific stuff.
> It's easy to add/remove/rename attributes, create additional tables and 
> so on.
> And although it's easier for me to develop on MySQL it should be easy to 
> switch to Derby.
> In the future when performance matters it could be successive replaced 
> by optimized queries.
> 
> Joachim

I agree,
Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MessageRepository and key, msn, uid, Flags (Was: IMAP development)

[Joachim Draeger <jd...@gmx.de>]

Stefano Bagnara schrieb:

> Ok, I know I'm ignorant about IMAP. Sorry for so much questions!

No problem. :-)

> KEY => A unique string that identify a single message in the repository: 
> persistent over sessions and restart, right?

Right. Like it's used for pop3.

> UID => A unique number that identify a single message in the repository: 
> persistent over sessions and restart as long as UIDVALIDITY does not 
> change (can UIDVALIDITY be changed in the middle of a session?)

The more important point: the UID of an appended message has to be higher then 
the one before.
Normally: UID=1 for the first Message, UID=2 for the second...
If there are uids 1,2,3 and 3 is deleted next appended is 4.
So you have to save the last appended uid somewhere.

Normally: UIDVALIDITY is created randomly at mailbox creation and never changes.
The client remembers UIDVALIDITY just to check if the UIDs it has seen in the 
last sessions are still valid. If UIDVALIDITY has changed, the mailbox has been 
probably deleted and recreated in the meantime.

> MSN => Are temporary numbers having meanings only inside a given session 
> and uniquely associated to an underlying message (From my understanding 
> we cannot use UIDs as MSN because MSN must start with 1 and be a 
> complete sequence, is this right?).

Right. Even if a message gets externally deleted during the session, the numbers 
have to be stable. Accessing the deleted message leads to an error.

> So MSN is the message number that we also use when deleting messages by 
> message number in pop3. 

Right. The concept is the same in IMAP, POP3 and JavaMail.

> What about POP3 UIDL ? POP3 require persistent 
> keys: can we use UID or do we need KEYs for this?
> I would like to understand the differences between UID and KEY and why 
> we need both. Can't we have one of them to implement the other?

KEY = POP3 UIDL is not required by IMAP.
An implementation that is based on UID can simply return a String representation 
  of the UID as the KEY.

The idea is that there may be repositories that don't support UIDs. (like todays 
MessageRepositories or pure Maildir or MBox) They could at least provide keys by 
a hash-sum.
Trying to access them by IMAP will raise an error.
Services that don't need UID capabilities (POP3) should always use KEYs so they 
are able to access every Mailbox.

>> You always have to check things like: is the mailbox/message still there?
>> Maybe you have to reopen resources do hashing/cashing/parsing each 
>> request...
>> You can't use a efficient SQL JOIN to fetch everything requested at once.
>  >
>> I noticed that you know mostly exactly what you want, e.g.: flags, 
>> MimeMessage, Uid
>> So the backend *can* optimize this request to fetch everything at once.
>> And if you just want stream and flags, it should provide you with 
>> stream and flags and not an all-purpose pseudo-light-weight 
>> MimeMessage monster. ;-)
>> And not require you to do a request for every single item.
> 
> Can't we load everything but the mimemessage and keep the lazy loading 
> (MimeMessageSource) for this one?

Well a MimeMessage wouldn't be sufficient because it doesn't provide the uid or 
the key. So you still need a result object.
MimeMessage is not extensible to provide additional attributes.

> Is it so much work to load flags and uid for each message? Can't we do 
> this as a first step, and optimize things later?

I said "the backend *can* optimize the request". It does not forbid to load 
everything at once. It does not forbid a lazy-loading MimeMessage.
But it does allow future optimizations.

>>> 2) Why Key/Flags/Msn/Uid/InternalDate are present in this interface 
>>> and, as an example, the From or the Subject is not there? Is this 
>>> related to what the IMAP protocol needs?
>>
>> Oh yes, I thought about requesting headers without MimeMessage 
>> monster, which maybe prepares resources to fetch a body stream...  :-)
>> And no: key is what James needs, msn is what pop3+imap needs
>> And Yes, the rest is needed by IMAP.
> 
> Remember that we already have a MimeMessageWrapper: we could work on it 
> to know what is already loaded and what will need to load the headers or 
> the body. We already do something similar (We disabled the header 
> lazyness in 2.3 because of a bug that I have to investigate yet).

But all that is just more complicated then just telling the backend what you 
would like to have.
A MessageResult is like a transaction: I don't have to fear that while dealing 
with header lines the flags are getting deleted.
I'm not proposing to abandon MimeMessageWrapper and lazy-loading. I just can't 
see the advantage to create a MimeMessage instance when I just want to have the 
size.
To have a simple Bean to provide requested results is IMO the "cleanest" solution.

> Tell me something more about flags: can we simplify them as key=>value 
> as string=>string pairs? Or what else?

Instead of using the JavaMail Flags object? I don't like it so much either.
There is a defined set of flags (ANSWERED, DELETED, DRAFT, FLAGGED, SEEN, RECENT)
So you could use bitwise integer values, boolean values, the names or abbreviations.

Joachim


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MessageRepository and key, msn, uid, Flags (Was: IMAP development)

[Stefano Bagnara <ap...@bago.org>]

Noel J. Bergman wrote:
>> Imo we should remove this concept of SPOOL from the message repository.
>> If one will ever want to implement a spool repository based on a
>> messagereposirory it will have to store the Mail informations in a
>> messagerepository way (as Flags, or as custom MimeHeaders).
>> MessageRepository should not have dependencies on the Mail object.
> 
> I'll try to catch up on the avalanche of JAMES e-mail later, but with
> respect to this, the plan for JAMES is to separate MAIL and MESSAGE.  And we
> might even separate SPOOL from MAIL.  I'm focusing on the stuff that Norman
> needs now, and will deal with normalized relational schema later.
> 
> 	--- Noel

I think we simply don't need MAIL. We just need SPOOL, and SPOOL make 
use of MESSAGE. The envelope informations are needed while we are 
spooling it, but not once we have the message in the user inbox (either 
POP3 or IMAP).

Imho we can do this simplification safely: the SpoolRepository is an 
extension of MailRepository, so if we'll ever need to use a 
MailRepository we can simply use a SpoolRepository instead.

Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MessageRepository and key, msn, uid, Flags (Was: IMAP development)

[Norman Maurer <nm...@byteaction.de>]

Stefano Bagnara schrieb:
> Noel J. Bergman wrote:
>>> Imo we should remove this concept of SPOOL from the message repository.
>>> If one will ever want to implement a spool repository based on a
>>> messagereposirory it will have to store the Mail informations in a
>>> messagerepository way (as Flags, or as custom MimeHeaders).
>>> MessageRepository should not have dependencies on the Mail object.
>>
>> I'll try to catch up on the avalanche of JAMES e-mail later, but with
>> respect to this, the plan for JAMES is to separate MAIL and MESSAGE.  
>> And we
>> might even separate SPOOL from MAIL.  I'm focusing on the stuff that 
>> Norman
>> needs now, and will deal with normalized relational schema later.
>>
>>     --- Noel
>
> I think we simply don't need MAIL. We just need SPOOL, and SPOOL make 
> use of MESSAGE. The envelope informations are needed while we are 
> spooling it, but not once we have the message in the user inbox 
> (either POP3 or IMAP).
>
> Imho we can do this simplification safely: the SpoolRepository is an 
> extension of MailRepository, so if we'll ever need to use a 
> MailRepository we can simply use a SpoolRepository instead.
>
> Stefano 

+1
Norman



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

RE: MessageRepository and key, msn, uid, Flags (Was: IMAP development)

["Noel J. Bergman" <no...@devtech.com>]

> Imo we should remove this concept of SPOOL from the message repository.
> If one will ever want to implement a spool repository based on a
> messagereposirory it will have to store the Mail informations in a
> messagerepository way (as Flags, or as custom MimeHeaders).
> MessageRepository should not have dependencies on the Mail object.

I'll try to catch up on the avalanche of JAMES e-mail later, but with
respect to this, the plan for JAMES is to separate MAIL and MESSAGE.  And we
might even separate SPOOL from MAIL.  I'm focusing on the stuff that Norman
needs now, and will deal with normalized relational schema later.

	--- Noel



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org