Let's Create An Alternative To MIMEMessage

[robert burrell donkin <ro...@gmail.com>]

i've been thinking for a long while of using a replacement for
MIMEMessage as an intermediary representation in IMAP (and possibly
elsewhere in JAMES). i know that stefano advocates using a lazy
loading MimeMessage but i've taken a good look at this and IMHO this
is too much effort when it comes to IMAP: tuning a MimeMessage for
IMAP is too inefficient (fixing the buggy parsing would require
reparsing the message).

so, i'd like to propose that JAMES looks for an alternative to
MIMEMessage for server side work

requirements:
1. fully interface based API (not a subclassing framework)
2. equal billing for nio and stream io
3. first class support for raw access to bytes, second class for
chars. third class support for object representations (if at all)
4. lightweight
5. easy partial and lazy parsing
6. first class support for bytebuffer storage
7. support for IMAP requirements such as line and byte counts for parts
8. support for meta-data
9. support for outputting components of the message
10. decoupled from javamail

opinions?

have i missed anything?

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Let's Create An Alternative To MIMEMessage

[robert burrell donkin <ro...@gmail.com>]

On 7/12/07, Stefano Bagnara <ap...@bago.org> wrote:
> robert burrell donkin ha scritto:
> >> > 6. first class support for bytebuffer storage
> >> > 7. support for IMAP requirements such as line and byte counts for parts
> >>
> >> charser/encoding conversions would be needed for ESMTP/8BITMIME support.
> >
> > 8BITMIME is an IMAP extension
> >
> > from an nio perspective, both charset and encoding would be issues
> > best left to the application. of course this then raises the
> > performance issue of executing the same decoding multiple times.
>
> I don't think that multiple conversions is a big deal. Btw we can do
> some optimization by always keep the last conversion and change only if
> strictly necessary.
> E.g: if we receive 8bitmime mail and we have to deliver via 8bitmime
> server then we can avoid conversion. If we receive 7bitmime mail we can
> avoid conversions at all.

ok

> > not sure i have an easy answer for this
> >
> >> > 8. support for meta-data
> >>
> >> can you elaborate?
> >
> > for example, JAMES attributes and javamail flags
>
> Why does this need to be implemented IN the MimeMessage handling
> structure? is this something that the MIME specification take into
> consideration? Can't we keep the Envelope/MimeMessage paradigm and place
> meta data in the envelopes ?
> This is not to say I'm against metadata, just to better understand the
> motivation behind the requirement.

that's a good question :-)

i suppose that this is also a question about API design within JAMES:
what objects do we pass to represent mail?

> >> > 9. support for outputting components of the message
> >>
> >> So, what do you keep in memory? the mime structure? And if possible
> >> pointers to streams for every parts (possibly shared) so that you can
> >> lazy parse them when you try to output them using a different encoding?
> >
> > when using a bytebuffer, everything is already in memory so this is
> > easy. doesn't sound very easy for streaming solutions though.
>
> How do we deal with 1GB mime messages?

ATM IMAP handles them badly

the torque backend stores these as a byte array. io is done via
streams. in between, the data is transfered as MimeMessage and string
buffer.

in general, this kind of parser would typically work on temporary
files. using nio, the large messages would be saved directly by the OS
to disc on the way in. the parser would then work on a memory mapped
buffer.

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Let's Create An Alternative To MIMEMessage

[Stefano Bagnara <ap...@bago.org>]

robert burrell donkin ha scritto:
>> > 6. first class support for bytebuffer storage
>> > 7. support for IMAP requirements such as line and byte counts for parts
>>
>> charser/encoding conversions would be needed for ESMTP/8BITMIME support.
> 
> 8BITMIME is an IMAP extension
> 
> from an nio perspective, both charset and encoding would be issues
> best left to the application. of course this then raises the
> performance issue of executing the same decoding multiple times.

I don't think that multiple conversions is a big deal. Btw we can do
some optimization by always keep the last conversion and change only if
strictly necessary.
E.g: if we receive 8bitmime mail and we have to deliver via 8bitmime
server then we can avoid conversion. If we receive 7bitmime mail we can
avoid conversions at all.

> not sure i have an easy answer for this
> 
>> > 8. support for meta-data
>>
>> can you elaborate?
> 
> for example, JAMES attributes and javamail flags

Why does this need to be implemented IN the MimeMessage handling
structure? is this something that the MIME specification take into
consideration? Can't we keep the Envelope/MimeMessage paradigm and place
meta data in the envelopes ?
This is not to say I'm against metadata, just to better understand the
motivation behind the requirement.

>> > 9. support for outputting components of the message
>>
>> So, what do you keep in memory? the mime structure? And if possible
>> pointers to streams for every parts (possibly shared) so that you can
>> lazy parse them when you try to output them using a different encoding?
> 
> when using a bytebuffer, everything is already in memory so this is
> easy. doesn't sound very easy for streaming solutions though.

How do we deal with 1GB mime messages?

>> > 10. decoupled from javamail
>>
>> +1 Maybe the resulting framework could be used to create an alternative
>> backend for javamail for legacy applications, but this is too far in the
>> future.
> 
> just at the thinking stage ATM
> 
>> I think all of this belongs to the mime4j project.
>> Do you plan to start working on something from mime4j code?
> 
> not sure
> 
> it sounds like there may well be issues in creating a general solution
> suitable for both streaming and bytebuffer backends. it may be that
> they are just too different.
> 
> i'll probably protocol a clean API with an nio-centric MIME parser for
> IMAP. if something DOM-like would be a useful complement to the
> streaming SAX and (hopefully) pull parsers in Mime4J then we can tidy
> it up and discuss the optimum general solution later.

+1

> i would like to expose more options for alternative parsers (as well
> as MIMEMessage) through the mailbox interface but i'll start another
> thread for that.
> 
> - robert

Thank you,
Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Let's Create An Alternative To MIMEMessage

[robert burrell donkin <ro...@gmail.com>]

On 7/11/07, Stefano Bagnara <ap...@bago.org> wrote:
> robert burrell donkin ha scritto:
> > i've been thinking for a long while of using a replacement for
> > MIMEMessage as an intermediary representation in IMAP (and possibly
> > elsewhere in JAMES). i know that stefano advocates using a lazy
> > loading MimeMessage but i've taken a good look at this and IMHO this
> > is too much effort when it comes to IMAP: tuning a MimeMessage for
> > IMAP is too inefficient (fixing the buggy parsing would require
> > reparsing the message).
>
> To clarify my position: I hate javamail ;-). BUT I often complained some
> common javamail FUD because I had to learn javamail internal very well
> to have a somehow working MimeMessageWrapper/CoWProxy in JAMES Server.

yeh - nice work, that

> What I say is that often MimeMessage is extensible enough to apply
> optimizations and more, the problem is that no one did that and that
> doing this in javamail is much more difficult than creating from
> scratch. On the other side, if you create new interface you bind your
> code to yet another interface.

yes, it's a tough choice

it is extensible but it is a subclassing framework and this imposes
costs. it is read-write and input stream based whereas IMAP needs
something quick, read-only and nio-based

> That said, I think that Mime4J can be used and your mimemessage needs
> should be solved in mime4j and used by JAMES Server.

maybe so

> > so, i'd like to propose that JAMES looks for an alternative to
> > MIMEMessage for server side work

<snip>

> > 5. easy partial and lazy parsing
>
> A problem we may have is for shared streams accessed by multiple threads
> at the same time: javamail has the concept of SharedInputStream
> interface for this to be performant and avoid wasting resources.
> Do you plan to support something similar?

this is an example of complex that is unnecessary from the nio
perspective: the contents of a read only bytebuffer can easily be
share by as many threads as necessary.

i can see that this would be tricky when dealing with a stream.

> > 6. first class support for bytebuffer storage
> > 7. support for IMAP requirements such as line and byte counts for parts
>
> charser/encoding conversions would be needed for ESMTP/8BITMIME support.

8BITMIME is an IMAP extension

from an nio perspective, both charset and encoding would be issues
best left to the application. of course this then raises the
performance issue of executing the same decoding multiple times.

not sure i have an easy answer for this

> > 8. support for meta-data
>
> can you elaborate?

for example, JAMES attributes and javamail flags

> > 9. support for outputting components of the message
>
> So, what do you keep in memory? the mime structure? And if possible
> pointers to streams for every parts (possibly shared) so that you can
> lazy parse them when you try to output them using a different encoding?

when using a bytebuffer, everything is already in memory so this is
easy. doesn't sound very easy for streaming solutions though.

> > 10. decoupled from javamail
>
> +1 Maybe the resulting framework could be used to create an alternative
> backend for javamail for legacy applications, but this is too far in the
> future.

just at the thinking stage ATM

> I think all of this belongs to the mime4j project.
> Do you plan to start working on something from mime4j code?

not sure

it sounds like there may well be issues in creating a general solution
suitable for both streaming and bytebuffer backends. it may be that
they are just too different.

i'll probably protocol a clean API with an nio-centric MIME parser for
IMAP. if something DOM-like would be a useful complement to the
streaming SAX and (hopefully) pull parsers in Mime4J then we can tidy
it up and discuss the optimum general solution later.

i would like to expose more options for alternative parsers (as well
as MIMEMessage) through the mailbox interface but i'll start another
thread for that.

- robert

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Let's Create An Alternative To MIMEMessage

[Stefano Bagnara <ap...@bago.org>]

robert burrell donkin ha scritto:
> i've been thinking for a long while of using a replacement for
> MIMEMessage as an intermediary representation in IMAP (and possibly
> elsewhere in JAMES). i know that stefano advocates using a lazy
> loading MimeMessage but i've taken a good look at this and IMHO this
> is too much effort when it comes to IMAP: tuning a MimeMessage for
> IMAP is too inefficient (fixing the buggy parsing would require
> reparsing the message).

To clarify my position: I hate javamail ;-). BUT I often complained some
common javamail FUD because I had to learn javamail internal very well
to have a somehow working MimeMessageWrapper/CoWProxy in JAMES Server.

What I say is that often MimeMessage is extensible enough to apply
optimizations and more, the problem is that no one did that and that
doing this in javamail is much more difficult than creating from
scratch. On the other side, if you create new interface you bind your
code to yet another interface.

That said, I think that Mime4J can be used and your mimemessage needs
should be solved in mime4j and used by JAMES Server.

> so, i'd like to propose that JAMES looks for an alternative to
> MIMEMessage for server side work
> 
> requirements:
> 1. fully interface based API (not a subclassing framework)

+1 (even if subclassing has its advantages I also prefer the advantages
of interface based frameworks).

> 2. equal billing for nio and stream io

+1

> 3. first class support for raw access to bytes, second class for
> chars. third class support for object representations (if at all)

Right, we probably don't need object representations.

> 4. lightweight

no comment.

> 5. easy partial and lazy parsing

A problem we may have is for shared streams accessed by multiple threads
at the same time: javamail has the concept of SharedInputStream
interface for this to be performant and avoid wasting resources.
Do you plan to support something similar?

> 6. first class support for bytebuffer storage
> 7. support for IMAP requirements such as line and byte counts for parts

charser/encoding conversions would be needed for ESMTP/8BITMIME support.

> 8. support for meta-data

can you elaborate?

> 9. support for outputting components of the message

So, what do you keep in memory? the mime structure? And if possible
pointers to streams for every parts (possibly shared) so that you can
lazy parse them when you try to output them using a different encoding?

> 10. decoupled from javamail

+1 Maybe the resulting framework could be used to create an alternative
backend for javamail for legacy applications, but this is too far in the
future.

> opinions?
> 
> have i missed anything?
> 
> - robert

I think all of this belongs to the mime4j project.
Do you plan to start working on something from mime4j code?

Stefano


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Let's Create An Alternative To MIMEMessage

[Jochen Wiedmann <jo...@gmail.com>]

robert burrell donkin-2 wrote:
> 
> 4. lightweight
> 5. easy partial and lazy parsing
> 6. first class support for bytebuffer storage
> 

Did you note MIME4J-19? Might help.

Jochen
-- 
View this message in context: http://www.nabble.com/Let%27s-Create-An-Alternative-To-MIMEMessage-tf4064240.html#a11548594
Sent from the James - Dev mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org