Re: MailboxManager API and session orientation was: Jsieve Configuration

[Joachim Draeger <jd...@joachim-draeger.de>]

Am Mittwoch, den 10.01.2007, 20:50 +0000 schrieb robert burrell donkin:

> > > > From the other side developers maybe just don't need a full featured
> > > > ImapSession for their needs and want to use an easier interface.
> > > >
> > > > That was the intention for providing different "flavors" of sessions.
> > > > MailboxSession easy to ImapMailboxSession advanced.
> > > >
> > > > At the moment I'm not even sure if there should be different interfaces
> > > > and if they should be in one hierarchy or separated... It's all
> > > > possible. :-)
> > >
> > > ATM the API is very session oriented. to access email means opening a
> > > session with state preserved between invocations. i can appreciate
> > > that with stateful protocols, it's going to be necessary to have
> > > really pretty different session objects.
> >
> > The session orientation is not only for session orientated protocols.
> > It's also needed for many backends.
> 
> -1
> google REST :-)

What has it to do with accessing specific backends?

I guess another good example are session orientated backends like
JavaMail stores. If you want to access them stateless it becomes a mess.
 
> > Think of a mbox and pop3. The backend opens the file and indexes it.
> 
> that is not conversational state but an optimization

Right, but without a session there is no possibility for that
optimization. And in fact it has to be done for many backends. 
If not, fine. Just do nothing on session opening. 
What can be done stateful and what stateless depends on the backend and
the implementation. 
If you take some operations and define them as stateless in the API, a
backend implementation has no possibility to optimize any stateful
thing.
If you define it inside a session, the implementation can just pass it
through, without anything heavyweight. 

> > It should keep the index as long as the pop3 session exists.
> > The expunge of deleted messages should be done when the pop3 session
> > closes.
> 
> POP3 is not a stateless protocol
> 
> the current generation of email protocols are broken. new protocols
> are needed for the internet :-)

Right, I'm talking about an API that performs well for SMTP, POP3 and
IMAP. That is the goal. Future API for stateless protocols is a another
discussion.

> > Transactions mentioned by Stefano are also a point.
> 
> what do transactions have to do with conversational state?

Transactions are currently not part of the API. It is still to be
researched what has to be attended when e.g. moving mails from spool to
backend. But that's really another story.

> the easiest way to kill performance is by long running transactions

So the goal will be having short running smart transactions.

> > In fact sessions bring a bit inconvenience for the developer. But in the
> > past stateless MailRepositories made problems. They were solved by
> > putting it into a weak hash map. It's no good practice to let the GC
> > decide about sessions. :-)
> 
> IMHO one of the problems with the current API is a lack of clarity about state
> 
> some state is conversational, other state is more naturally associated
> with the actual mailbox and could be reasonably shared between
> clients.

What do you have in mind?

IMO the implementation, that was written for a certain backend with a
certain purpose, can decide perfectly how to handle this.

> > That's like it is working now. The whole message number stuff is done
> > inside a wrapper on top on a stateless backend class.
> 
> the problem with using wrappers is that commands can't easily access
> the lighter stateless layer

Heavy and lighter depends on the implementation. If a wrapper just
passes through, there is no reasonable cost.
IMO it is important that the session implementation has the power to
decide what to pass through and when to do any additional operations. 

> > > > It's really difficult to make the API general but still allow
> > > > optimizations for a well performing IMAP server that is able to serve
> > > > *many* clients.
> > >
> > > +1
> > >
> > > it's hard to see the right way to factor the code but IMHO it's worth
> > > making the effort
> >
> > That's my intention and again: I hope for some input on concrete issues.
> > My last monologues about this topic always ended in some kind of a
> > general discussion. Always some protocol agnostic / aware remarks... :-)
> 
> but notice this time that it started from the general and is now
> working towards the specific :-)

Is it? Most of current discussion are universal phrases with outlook on
future protocols, turning round in circles.
Concrete is talking about method A in interface B. Or proposing
completely modified interfaces to compare one against the other.

> the inheritance hierarchy seems overly deep to me (both in the
> interfaces and the implementation). perhaps this is a sign that the
> API would benefit from additional layering.

Can you elaborate additional layering?

Joachim






---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[robert burrell donkin <ro...@gmail.com>]

On 1/11/07, Joachim Draeger <jd...@joachim-draeger.de> wrote:
> Am Mittwoch, den 10.01.2007, 20:50 +0000 schrieb robert burrell donkin:

<snip>

> > > In fact sessions bring a bit inconvenience for the developer. But in the
> > > past stateless MailRepositories made problems. They were solved by
> > > putting it into a weak hash map. It's no good practice to let the GC
> > > decide about sessions. :-)
> >
> > IMHO one of the problems with the current API is a lack of clarity about state
> >
> > some state is conversational, other state is more naturally associated
> > with the actual mailbox and could be reasonably shared between
> > clients.
>
> What do you have in mind?

the classic recipe is to store state in the most appropriate place and
then pass the state backwards as additional input parameters. this
allows each layer to operate more and more concurrently with less and
less state.

> IMO the implementation, that was written for a certain backend with a
> certain purpose, can decide perfectly how to handle this.

there are different ways to achieve this goal

the current API is overly large and coupled. it is perfectly possible
for two tightly coupled systems to operate on either side of a
reasonably general API. (usually means switching to a  messaging style
API.)

<snip>

> > the inheritance hierarchy seems overly deep to me (both in the
> > interfaces and the implementation). perhaps this is a sign that the
> > API would benefit from additional layering.
>
> Can you elaborate additional layering?

not sure: it's just usually the right refactoring when faced with this
situation :-)

it's hard for understand an API with so much inheritance and so many
small interfaces.

perhaps a protocol layer somewhere might help or maybe a more distinct
mailbox layer

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[robert burrell donkin <ro...@gmail.com>]

On 1/13/07, Stefano Bagnara <ap...@bago.org> wrote:
> robert burrell donkin wrote:
> > manage the state more clearly. allow the backend access to more
> > information about the conversational state. the parsing layer should
> > manage the conversational state but this should be done through an
> > interface. the implementation should be provided by the backend
> > allowing additional information to be associated.
> >
> > the whole IMAP API interface would be something like:
> >
> > public IMAPConversationalSession createIMAPSession()
> > public void execute(IMAPConversationalSession state, IMAPCommand command)
>
> Robert, I'm not sure I understand you.
> Are you proposing an IMAP API composed by 2 methods? A command pattern
> applied to a session? Do you call it an API?

i most definitely call it an API  :-)

one of the messaging style

> I think that this is too abstract and I feel I'm not understanding what
> you propose.
>
> If I understand you simply want to change the current pattern:
> public IMAPConversationalSession createIMAPSession()
> public void IMAPConversationalSession.execute(IMAPCommand command)
>
> So simply have the session as a parameter for an executor instead of
> using it as the executor.

the concept of what a session is changed too: it's conversational session

> If I understood you you simply propose to add
> a Memento pattern to the Command pattern.

yep - i never claimed to be original

> Imho this is a small change
> and can be changed with small refactorings, but I don't see HOW this
> changes the complexity of the API.

it's small but subtle

it changes the conceptual relationships between the layers

the IMAP server manages conversational state, parses input and formats output

the intermediary layer processes the commands and interacts with the
data store. this ensure that all information that is know is available
to the implementation.

the interface to the data store at this layer may well become a more
usual call-centric API but it's pushed to a lower level in the API
structure and allows both radically different implementations, and
implementations that provide specific optimizations targeting backends
known to allow them.

> > yes, i know that this pushes some of the problems into the backend but
> > i think that these could be solved by layering below the mailbox
> >
> > - robert
>
> If you can provide some sort of code to explain your ideas I would be
> really happy.
> Feel free to create a sandbox and provide a strawman implementation of
> what you mean or to refactor the real code (much better to understand).
> If you feel more confortable with diagrams I'm happy with them too, but
> I need something more concrete to continue this discussion.

if there's interest, maybe i'll pull together some UML

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[Stefano Bagnara <ap...@bago.org>]

robert burrell donkin wrote:
> manage the state more clearly. allow the backend access to more
> information about the conversational state. the parsing layer should
> manage the conversational state but this should be done through an
> interface. the implementation should be provided by the backend
> allowing additional information to be associated.
> 
> the whole IMAP API interface would be something like:
> 
> public IMAPConversationalSession createIMAPSession()
> public void execute(IMAPConversationalSession state, IMAPCommand command)

Robert, I'm not sure I understand you.
Are you proposing an IMAP API composed by 2 methods? A command pattern 
applied to a session? Do you call it an API?

I think that this is too abstract and I feel I'm not understanding what 
you propose.

If I understand you simply want to change the current pattern:
public IMAPConversationalSession createIMAPSession()
public void IMAPConversationalSession.execute(IMAPCommand command)

So simply have the session as a parameter for an executor instead of 
using it as the executor. If I understood you you simply propose to add 
a Memento pattern to the Command pattern. Imho this is a small change 
and can be changed with small refactorings, but I don't see HOW this 
changes the complexity of the API.

> yes, i know that this pushes some of the problems into the backend but
> i think that these could be solved by layering below the mailbox
> 
> - robert

If you can provide some sort of code to explain your ideas I would be 
really happy.
Feel free to create a sandbox and provide a strawman implementation of 
what you mean or to refactor the real code (much better to understand).
If you feel more confortable with diagrams I'm happy with them too, but 
I need something more concrete to continue this discussion.

Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[robert burrell donkin <ro...@gmail.com>]

On 1/13/07, Joachim Draeger <jd...@joachim-draeger.de> wrote:
> Am Samstag, den 13.01.2007, 10:53 +0000 schrieb robert burrell donkin:
>
> > > > this is a good example of why i claim that state needs to considered
> > > > more carefully. hopefully it may also illustrate why i claim that
> > > > session is harmful.
> > >
> > > That's not fair. :-( It's just an example for work-in-progress. I
> > > explained that I removed the lazy init because there was a problem I
> > > hadn't time to investigate further on.
> >
> > didn't intend this to be a criticism of you personally - i understand
> > how difficult IMAP is and think you've been doing a great job :-)
>
> > i shouldn't have rushed to specifics so quickly since i'm trying to
> > criticize the API rather than the implementation
> >
> > but it is a good illustration of the kinds of problems that the
> > current API creates
>
> No, I don't agree. You can not judge missing optimization of a prototype
> implementation as weak points of the API.

the API fails to pass sufficient information to the backend to allow
optimization. this is an architectural flaw. yes, a good
implementation will find a work around but the price is messy code.

> And please note that concrete proposals and suggestions for improvements
> have much more value than critique and abstract discussion.
>
> Critique is just more welcome when it comes with a concrete solution
> like a patch. :-)

going to the trouble of locally ripping out and replacing the current
APIs seems like a waste of time to me and seems likely just to cause
heated debates in the future. i can sketch a solution, though.

the current APIs are a classic example of an API which has grown over
the years without design. they tend to become crufty as every new
optimization or function requires more methods to be added. each extra
call breaks backward compatibility so it's not really possible to have
more than one implementation.

move to a cleaner, simpler interface. the command pattern is a nature
fit. let the commands handle the parsing and the backend the data
processing.

manage the state more clearly. allow the backend access to more
information about the conversational state. the parsing layer should
manage the conversational state but this should be done through an
interface. the implementation should be provided by the backend
allowing additional information to be associated.

the whole IMAP API interface would be something like:

public IMAPConversationalSession createIMAPSession()
public void execute(IMAPConversationalSession state, IMAPCommand command)

yes, i know that this pushes some of the problems into the backend but
i think that these could be solved by layering below the mailbox

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[Joachim Draeger <jd...@joachim-draeger.de>]

Am Samstag, den 13.01.2007, 10:53 +0000 schrieb robert burrell donkin:

> > > this is a good example of why i claim that state needs to considered
> > > more carefully. hopefully it may also illustrate why i claim that
> > > session is harmful.
> >
> > That's not fair. :-( It's just an example for work-in-progress. I
> > explained that I removed the lazy init because there was a problem I
> > hadn't time to investigate further on.
> 
> didn't intend this to be a criticism of you personally - i understand
> how difficult IMAP is and think you've been doing a great job :-)

> i shouldn't have rushed to specifics so quickly since i'm trying to
> criticize the API rather than the implementation
> 
> but it is a good illustration of the kinds of problems that the
> current API creates

No, I don't agree. You can not judge missing optimization of a prototype
implementation as weak points of the API. 

And please note that concrete proposals and suggestions for improvements
have much more value than critique and abstract discussion. 

Critique is just more welcome when it comes with a concrete solution
like a patch. :-)

> messaging number is state that the current API forces to be associated
> with the mailbox session since it's the only available option.
> 
> there are different approaches to the design of APIs and to state
> management that i think may be worth considering. IMHO the system
> would also benefit from another look at the layering.
> 
> interested?

Yes under following conditions: Concrete, by providing interfaces and
code as samples for a discussion base. 
Separate discussion on current use cases (SMTP/IMAP/POP3, many different
possible backends) from future ones.

Difficult backends are JavaMail stores, MBox, Maildir with separate
metadata file or even a virtual one through IMAP.

Joachim




---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[robert burrell donkin <ro...@gmail.com>]

On 1/11/07, Joachim Draeger <jd...@joachim-draeger.de> wrote:
> Am Donnerstag, den 11.01.2007, 20:14 +0000 schrieb robert burrell
> donkin:
> > On 1/11/07, Joachim Draeger <jd...@joachim-draeger.de> wrote:
> > > Am Donnerstag, den 11.01.2007, 18:55 +0000 schrieb robert burrell
> > > donkin:
> > >
> > > > > > > Think of a mbox and pop3. The backend opens the file and indexes it.
> > > > > >
> > > > > > that is not conversational state but an optimization
> > > > >
> > > > > Right, but without a session there is no possibility for that
> > > > > optimization.
> > > >
> > > > there are different ways which this could be implemented. the current
> > > > session concept is not necessary and in some cases can be detrimental.
> > > >
> > > > a flaw with the current API is that a session is opened before the
> > > > factory has any idea about what's it's to be asked to do. it is
> > > > therefore inefficient to perform any optimization when the session is
> > > > opened.
> > > >
> > > > for example, whenever an IMAP session is opened the session
> > > > pre-emptively caches the UIDs. at this time, the session cannot know
> > > > whether a particular command will need this information or not.
> > >
> > > As I said the last time: This is intended to be done on first use.
> > > Just lazy initialization on demand. That way you need not know whether
> > > it is needed or not.
> >
> > this is a good example of why i claim that state needs to considered
> > more carefully. hopefully it may also illustrate why i claim that
> > session is harmful.
>
> That's not fair. :-( It's just an example for work-in-progress. I
> explained that I removed the lazy init because there was a problem I
> hadn't time to investigate further on.

didn't intend this to be a criticism of you personally - i understand
how difficult IMAP is and think you've been doing a great job :-)

i shouldn't have rushed to specifics so quickly since i'm trying to
criticize the API rather than the implementation

but it is a good illustration of the kinds of problems that the
current API creates

messaging number is state that the current API forces to be associated
with the mailbox session since it's the only available option.

there are different approaches to the design of APIs and to state
management that i think may be worth considering. IMHO the system
would also benefit from another look at the layering.

interested?

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[Joachim Draeger <jd...@joachim-draeger.de>]

Am Donnerstag, den 11.01.2007, 20:14 +0000 schrieb robert burrell
donkin:
> On 1/11/07, Joachim Draeger <jd...@joachim-draeger.de> wrote:
> > Am Donnerstag, den 11.01.2007, 18:55 +0000 schrieb robert burrell
> > donkin:
> >
> > > > > > Think of a mbox and pop3. The backend opens the file and indexes it.
> > > > >
> > > > > that is not conversational state but an optimization
> > > >
> > > > Right, but without a session there is no possibility for that
> > > > optimization.
> > >
> > > there are different ways which this could be implemented. the current
> > > session concept is not necessary and in some cases can be detrimental.
> > >
> > > a flaw with the current API is that a session is opened before the
> > > factory has any idea about what's it's to be asked to do. it is
> > > therefore inefficient to perform any optimization when the session is
> > > opened.
> > >
> > > for example, whenever an IMAP session is opened the session
> > > pre-emptively caches the UIDs. at this time, the session cannot know
> > > whether a particular command will need this information or not.
> >
> > As I said the last time: This is intended to be done on first use.
> > Just lazy initialization on demand. That way you need not know whether
> > it is needed or not.
> 
> this is a good example of why i claim that state needs to considered
> more carefully. hopefully it may also illustrate why i claim that
> session is harmful.

That's not fair. :-( It's just an example for work-in-progress. I
explained that I removed the lazy init because there was a problem I
hadn't time to investigate further on.

Joachim







---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[robert burrell donkin <ro...@gmail.com>]

On 1/11/07, Joachim Draeger <jd...@joachim-draeger.de> wrote:
> Am Donnerstag, den 11.01.2007, 18:55 +0000 schrieb robert burrell
> donkin:
>
> > > > > Think of a mbox and pop3. The backend opens the file and indexes it.
> > > >
> > > > that is not conversational state but an optimization
> > >
> > > Right, but without a session there is no possibility for that
> > > optimization.
> >
> > there are different ways which this could be implemented. the current
> > session concept is not necessary and in some cases can be detrimental.
> >
> > a flaw with the current API is that a session is opened before the
> > factory has any idea about what's it's to be asked to do. it is
> > therefore inefficient to perform any optimization when the session is
> > opened.
> >
> > for example, whenever an IMAP session is opened the session
> > pre-emptively caches the UIDs. at this time, the session cannot know
> > whether a particular command will need this information or not.
>
> As I said the last time: This is intended to be done on first use.
> Just lazy initialization on demand. That way you need not know whether
> it is needed or not.

this is a good example of why i claim that state needs to considered
more carefully. hopefully it may also illustrate why i claim that
session is harmful.

each time that getImapMailboxSession is called on MailboxManager, a
new instance is created and the cache is filled. the selected session
is cached in the ImapSessionState (the conversational session, i
think) so it's true that commands which operate on the selected
mailbox pay this cost only once.

any command which does not operate on the selected mailbox is forced
to the price for this setup on each execution.

this is a common defect found in this type of API: when the
ImapMailboxSession is obtained the backend has no way of knowing what
it will be used for. the backend can therefore not perform smart
optimizations.

there are alternative approaches to this kind of design but they
involve some up front payments in terms of increased analysis and (as
i've amply demonstrated) i don't know anyway near enough about IMAP to
perform this alone. interested?

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[Joachim Draeger <jd...@joachim-draeger.de>]

Am Donnerstag, den 11.01.2007, 18:55 +0000 schrieb robert burrell
donkin:

> > > > Think of a mbox and pop3. The backend opens the file and indexes it.
> > >
> > > that is not conversational state but an optimization
> >
> > Right, but without a session there is no possibility for that
> > optimization.
> 
> there are different ways which this could be implemented. the current
> session concept is not necessary and in some cases can be detrimental.
> 
> a flaw with the current API is that a session is opened before the
> factory has any idea about what's it's to be asked to do. it is
> therefore inefficient to perform any optimization when the session is
> opened.
> 
> for example, whenever an IMAP session is opened the session
> pre-emptively caches the UIDs. at this time, the session cannot know
> whether a particular command will need this information or not.

As I said the last time: This is intended to be done on first use.
Just lazy initialization on demand. That way you need not know whether
it is needed or not. 

Joachim






---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: MailboxManager API and session orientation was: Jsieve Configuration

[robert burrell donkin <ro...@gmail.com>]

On 1/11/07, Joachim Draeger <jd...@joachim-draeger.de> wrote:
> Am Mittwoch, den 10.01.2007, 20:50 +0000 schrieb robert burrell donkin:

<snip>

> > > Think of a mbox and pop3. The backend opens the file and indexes it.
> >
> > that is not conversational state but an optimization
>
> Right, but without a session there is no possibility for that
> optimization.

there are different ways which this could be implemented. the current
session concept is not necessary and in some cases can be detrimental.

a flaw with the current API is that a session is opened before the
factory has any idea about what's it's to be asked to do. it is
therefore inefficient to perform any optimization when the session is
opened.

for example, whenever an IMAP session is opened the session
pre-emptively caches the UIDs. at this time, the session cannot know
whether a particular command will need this information or not.

> And in fact it has to be done for many backends.
> If not, fine. Just do nothing on session opening.

most data stores (if i understand you correctly) have notions of state
and session. i just think that the design would be cleaner and quicker
if the different notions of session were separated.

> What can be done stateful and what stateless depends on the backend and
> the implementation.

given appropriate design, it shouldn't do

> If you take some operations and define them as stateless in the API, a
> backend implementation has no possibility to optimize any stateful
> thing.

just pass the state through to the backend in the call. the backend
does not maintain the additional state but makes use of the data
passed in.

> If you define it inside a session, the implementation can just pass it
> through, without anything heavyweight.

the problem is that the session does not know about the protocol so it
cannot decide whether it a heavyweight or lightweight implementation
would be more appropriate.

> > > It should keep the index as long as the pop3 session exists.
> > > The expunge of deleted messages should be done when the pop3 session
> > > closes.
> >
> > POP3 is not a stateless protocol
> >
> > the current generation of email protocols are broken. new protocols
> > are needed for the internet :-)
>
> Right, I'm talking about an API that performs well for SMTP, POP3 and
> IMAP. That is the goal. Future API for stateless protocols is a another
> discussion.

stateless protocols scale. others do not. good state management is
crucial whether the protocol has conversational state or not.

> > > Transactions mentioned by Stefano are also a point.
> >
> > what do transactions have to do with conversational state?
>
> Transactions are currently not part of the API. It is still to be
> researched what has to be attended when e.g. moving mails from spool to
> backend. But that's really another story.
>
> > the easiest way to kill performance is by long running transactions
>
> So the goal will be having short running smart transactions.

the API should help not hinder this goal

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org