You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Stefano Bagnara <ap...@bago.org> on 2006/03/04 18:37:05 UTC
MimeMessage handling optimizations/changes for 2.3.0a2
Hi all,
Just spent 2 days studying javamail sourcecode and how James
creates/uses MimeMessages and I found that current javamail MimeMessage
(i worked on 1.3.2) has many protected methods and expose very much of
its internal behaviour.
Furthermore, you, perhaps, rememeber a long thread about the use of
SharedInputStream when creating new MimeMessages to avoid Javamail to
load the full message in memory.
I started writing tests for the MimeMessage and its various flavor
(depending on the constractor used) and for our own Wrappers:
MimeMessageWrapper/MimeMessageCopyOnWriteProxy.
I then found that most of the MimeMessageWrapper operations were the
exact "copy" of the MimeMessage behaviour and I though that I should
have tried to drop the "wrapping/delegation" behaviour in favor of a
simple extension.
This way it was simpler to define when the message is needed and I've
been able to avoid the message loading for most header operations
(previously needed). Furthermore, with this new approach, we don't need
to wrap all the MimeMessage methods and we'll have better compatibility
for the future.
I then changed every source of our messages to return SharedInputStreams
(starting from the Avalon stream repository) and I wrote a
SharedSequenceInputStream to handle sequence of sharedInputStreams to be
seen as a SharedInputStream.
The result is that all the tests still passes and that I removed a full
layer.
I then started working on the JDBCRepositories/MimeMessageJDBCSource to
remove the current ByteArray usage in favor of blob streaming and tested
it using latest Connecto/J drivers against a Mysql 4 db: the good news
is that this combo also handle messages bigger than the mysql
packet_size and the memory usage is at most the packet_size (the
connector/j allocate a buffer sized according to that parameter).
I made this behaviour optional by adding a new "getBody" option
(useBinaryStream) to the dbOptions in sqlResources so that testers could
try this feature over different dbs and provide feedback.
I think this could be a big improvement for performance and reduce the
memory footprint but I also think that this is a critical change that
will probably add more delay to a stable release.
My personal vote goes to commit the new code so that other
committers/developers could review the changes but I would like to know
what do you think about this, before moving.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
> Yes, I know that it's difficult, and that's why I asked about an
> approximate number.
> I thought you gave it some tries.
> You said that all your unit tests are passed with these new changes. If
> you used
> JUnit than you can have approximate timings too compared to the version
> without changes.
No time differences in the unit test run.
Our current tests are limited and they tests only small test messages so
there's no visible improvement.
I will try to add big message tests.
By now the main difference is that if I add a 20MB message to a spooling
test I'm able to run it with no OutOfMemory (and with no changes to the
default Xmx).
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Ahmed Mohombe <am...@yahoo.com>.
> Ahmed Mohombe wrote:
>>> I think this could be a big improvement for performance and reduce
>>> the memory footprint
>> Do you have and approximate numbers about the possible gain?
>
> It is difficult to give an estimate: totally depends on specific
> configurations/messages.
> Performance improvements are mainly provided by less memory usage (and
> GC usage) and by avoiding loading the message more times.
Yes, I know that it's difficult, and that's why I asked about an approximate number.
I thought you gave it some tries.
You said that all your unit tests are passed with these new changes. If you used
JUnit than you can have approximate timings too compared to the version without changes.
>> I (like many other users) would be much more happy with IMAP, config
>> reload without restarting JAMES, a better "Virtual Host"
>> support(something that just works), etc., but this is still sounds
>> fantastic IMHO.
>
> I'm sorry but I personally don't need IMAP/Virtual host/dynamic
> reloading so I prefer to spend my spare time improving things I use.
Sure, I understand. I don't use those things either, but I can't convince
most of the managers to use JAMES unless they'll be there - it's that simple :(.
> BTW I know that if we provide a better product we increase the
> possibilities to find new developers committed in new improvements, so
> I'm also working on the "virtual host" support.
And IMAP and config reload are also such features.
> I would also be glad to *help* on IMAP or other features but currently
> there is no one working on it and I don't want to be mantainer for code
> I don't even use.
Well, I still believe that unless JAMES doesn't have IMAP it can't really use that 'E' for
Enterprise in it's name.
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Ahmed Mohombe wrote:
>> I think this could be a big improvement for performance and reduce the
>> memory footprint
> Do you have and approximate numbers about the possible gain?
It is difficult to give an estimate: totally depends on specific
configurations/messages.
Performance improvements are mainly provided by less memory usage (and
GC usage) and by avoiding loading the message more times.
> I (like many other users) would be much more happy with IMAP, config
> reload without restarting JAMES, a better "Virtual Host"
> support(something that just works), etc., but this is still sounds
> fantastic IMHO.
I'm sorry but I personally don't need IMAP/Virtual host/dynamic
reloading so I prefer to spend my spare time improving things I use.
BTW I know that if we provide a better product we increase the
possibilities to find new developers committed in new improvements, so
I'm also working on the "virtual host" support.
I would also be glad to *help* on IMAP or other features but currently
there is no one working on it and I don't want to be mantainer for code
I don't even use.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Ahmed Mohombe <am...@yahoo.com>.
> The result is that all the tests still passes and that I removed a full
> layer.
Fantastic :).
> I think this could be a big improvement for performance and reduce the
> memory footprint
Do you have and approximate numbers about the possible gain?
> but I also think that this is a critical change that
> will probably add more delay to a stable release.
If the performance gain is good than it's not a problem.
I mean this release is already too late. A few months more or less makes no difference :).
> My personal vote goes to commit the new code so that other
> committers/developers could review the changes but I would like to know
> what do you think about this, before moving.
I personally think it's fantastic.
I (like many other users) would be much more happy with IMAP, config reload without restarting
JAMES, a better "Virtual Host" support(something that just works), etc., but this is still sounds
fantastic IMHO.
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Vincenzo Gianferrari Pini <vi...@praxis.it>.
Stefano,
great job!
I too think that it is worth to proceed with it.
Vincenzo
Noel J. Bergman wrote:
>>I though that I should have tried to drop the "wrapping/delegation"
>>behaviour in favor of a simple extension.
>>
>>
>
>I recall thinking the same. And moving to a shared inputstream and
>streaming should be a big win on footprint. Dan also mentioned it, with
>respect to Derby.
>
>
>
>>This way it was simpler to define when the message is needed and I've
>>been able to avoid the message loading for most header operations
>>(previously needed).
>>
>>
>
>This is a large part of what the wrapper was originally written to do, IIRC.
>Way before I started with JAMES.
>
>
>
>>The result is that all the tests still passes and that I removed a full
>>layer.
>>
>>
>
>
>
>>I made [streaming] optional by adding a new "getBody" option
>>(useBinaryStream) to the dbOptions in sqlResources so that
>>testers could try this feature over different dbs and provide
>>feedback.
>>
>>
>
>If you can complete the changes, I'd be in favor of getting them into the
>next milestone.
>
> --- Noel
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Stefano Bagnara" <ap...@bago.org>
To: "James Developers List" <se...@james.apache.org>
Sent: Tuesday, March 07, 2006 12:56 AM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
> Markus Kühn wrote:
>>> How different? I think you experience/work could help us following the
>>> right path...
>>
>> Exception handling is vital to me. It's more detailed. Something that
>> were not present in james 2.1.3.
>> It is only a modification based on my goals, nothing else. But I don't
>> think james is off the path in this respect.
>
> So this is somewhat similar to the current exception handling?
> http://wiki.apache.org/james/HandlingExceptions
Yes, but less options. I only support consumption of recipients by
matcher/mailets and mailets can specify a list of other mailets which the
message should ignore any time later. I don't think that is of general use.
>>>> JDBC mailstore classes can write the body of the message into the file
>>>> rep, as discussed before.
>>>
>>> Isn't this the same as using dbfile repositories in current james?
>>
>> No. I use db configuration with a modified JDBCSpoolRep and
>> JDBCMailStore.
>
> What are the difference if you also store the mail informations in the db
> and the message body in files?
Messages are stored entirely in the db, except those that exceed my size
limit; in
this case the body remains a file and is moved into the file rep.
> Have you implemented more efficient interfaces (move instead of
> store+remove) or what else?
Not interfaces. It's a different design. E. g. the 'incoming' spooler is a
new class, using only messages from the
handler (no loading from the db).
Most of the code was changed more or less. That's just because of the
different priorities, less flexibility more simplicity.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Markus Kühn wrote:
>> How different? I think you experience/work could help us following the
>> right path...
>
> Exception handling is vital to me. It's more detailed. Something that
> were not present in james 2.1.3.
> It is only a modification based on my goals, nothing else. But I don't
> think james is off the path in this respect.
So this is somewhat similar to the current exception handling?
http://wiki.apache.org/james/HandlingExceptions
>>> JDBC mailstore classes can write the body of the message into the
>>> file rep, as discussed before.
>>
>> Isn't this the same as using dbfile repositories in current james?
>
> No. I use db configuration with a modified JDBCSpoolRep and JDBCMailStore.
What are the difference if you also store the mail informations in the
db and the message body in files?
Have you implemented more efficient interfaces (move instead of
store+remove) or what else?
>> Did you removed avalon or you simply created your own "embedded avalon
>> container?
>
> The interfaces are the same, but renamed packages. So more the first one.
Thank you for the answers! It's always interesting to know what has been
done in "custom" branches of a project!
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Stefano Bagnara" <ap...@bago.org>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, March 06, 2006 8:18 PM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
>> Error handling in the Linearprocessors is different.
> How different? I think you experience/work could help us following the
> right path...
Exception handling is vital to me. It's more detailed. Something that were
not present in james 2.1.3.
It is only a modification based on my goals, nothing else. But I don't think
james is off the path in this respect.
>> JDBC mailstore classes can write the body of the message into the file
>> rep, as discussed before.
>
> Isn't this the same as using dbfile repositories in current james?
No. I use db configuration with a modified JDBCSpoolRep and JDBCMailStore.
> I'm interested expecially in the extensive error handling changes.
Back in 2003 javamail produced less descriptive error messages. Something I
filter out by searching the nested exceptions entirely.
For each message producing an error a protocol is generated; cause, time,
recipients are logged and saved for each try.
> Did you removed avalon or you simply created your own "embedded avalon
> container?
The interfaces are the same, but renamed packages. So more the first one.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Markus Kühn wrote:
> I have two spooler; one for incoming messages and another one for
> messages that produced an error.
I use also multiple spoolers and this is possible (very easy) with
custom assembly.xml on current james 2.3-dev.
> Error handling in the Linearprocessors is different.
How different? I think you experience/work could help us following the
right path...
> JDBC mailstore classes can write the body of the message into the file
> rep, as discussed before.
Isn't this the same as using dbfile repositories in current james?
> My version is designed for speed, simplicity and extensive error
> handling.
I'm interested expecially in the extensive error handling changes.
If I understood your points the speed/memory usage should be the same of
the current trunk after the changes I'm working on.
> Also I should be able to fix every problem in a reasonable
> time, so a dependency like phoenix was not an option.
Did you removed avalon or you simply created your own "embedded avalon
container" ?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Ahmed Mohombe <am...@yahoo.com>.
>>>> I also see no reason to use one as it doesn't deliver the most
>>>> important thing I would expect
>>>> from a container: "reloading the configuration without restarting
>>>> the server".
>>>
>>> Is this the *most* important thing?
>>> I think this is a personal interpretation of "most important thing"
>>> of a container.
>> It's not just "personal interpretation". If users ask why was a
>> container needed for JAMES the standard answer is *flexibility*. Form
>> a user perspective flexibility means exactly that.
>
> Maybe we're talking of different things: can you name a container
> supporting out of the box configuration reloadability for its component
> (with no full redeploy)?
No, we are not :). This is what a user understands under "flexibility" as a "container"
has no meaning for most of the users.
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Ahmed Mohombe wrote:
>>> I also see no reason to use one as it doesn't deliver the most
>>> important thing I would expect
>>> from a container: "reloading the configuration without restarting the
>>> server".
>>
>> Is this the *most* important thing?
>> I think this is a personal interpretation of "most important thing" of
>> a container.
> It's not just "personal interpretation". If users ask why was a
> container needed for JAMES the standard answer is *flexibility*. Form a
> user perspective flexibility means exactly that.
Maybe we're talking of different things: can you name a container
supporting out of the box configuration reloadability for its component
(with no full redeploy)?
Stefano
PS: avalon interfaces supports "alterable configurations", phoenix
itself does not support this avalon "option".
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Ahmed Mohombe <am...@yahoo.com>.
>> I also see no reason to use one as it doesn't deliver the most
>> important thing I would expect
>> from a container: "reloading the configuration without restarting the
>> server".
>
> Is this the *most* important thing?
> I think this is a personal interpretation of "most important thing" of a
> container.
It's not just "personal interpretation". If users ask why was a container needed for JAMES the
standard answer is *flexibility*. Form a user perspective flexibility means exactly that.
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Ahmed Mohombe wrote:
>> I don't use a container like phoenix.
> Is this possible? Is it possible at all to get rid of the container?
Sure.
> I also see no reason to use one as it doesn't deliver the most important
> thing I would expect
> from a container: "reloading the configuration without restarting the
> server".
Is this the *most* important thing?
I think this is a personal interpretation of "most important thing" of a
container.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
To: "James Developers List" <se...@james.apache.org>
Sent: Thursday, March 09, 2006 1:53 AM
Subject: RE: MimeMessage handling optimizations/changes for 2.3.0a2
> Markus Kühn wrote:
>> Noel J. Bergman wrote:
>> > I'd be inclined to migrate to OSGi at such time as we consider moving.
>> Are you currently involved in a OSGi project?
>
> We have several here at the ASF. Why do you ask?
I wanted to know how accepted OSGi is.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by "Noel J. Bergman" <no...@devtech.com>.
Markus Kühn wrote:
> Noel J. Bergman wrote:
> > I'd be inclined to migrate to OSGi at such time as we consider moving.
> Are you currently involved in a OSGi project?
We have several here at the ASF. Why do you ask?
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, March 06, 2006 9:32 PM
Subject: RE: MimeMessage handling optimizations/changes for 2.3.0a2
> We don't use that much of it, but I'd be inclined to migrate to OSGi at
> such
> time as we consider moving.
Are you currently involved in a OSGi project?
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by "Noel J. Bergman" <no...@devtech.com>.
> Btw I think that the avalon framework itself is a good thing to james
> and we could adapt james to run under j2ee or spring without getting
> rid of avalon interfaces.
We don't use that much of it, but I'd be inclined to migrate to OSGi at such
time as we consider moving.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Stefano Bagnara" <ap...@bago.org>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, March 06, 2006 8:13 PM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
> Btw I think that the avalon framework itself is a good thing to james and
> we could adapt james to run under j2ee or spring without getting rid of
> avalon interfaces.
I also would keep the avalon framework. It's not too complicated and it is
much less work :-).
The jee world can be reached with a handful of EJBs querying the stores of
james and also can easily produced with EJB 3.0 or XDoclet.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Ahmed Mohombe wrote:
>>>> I don't use a container like phoenix.
>>> Is this possible? Is it possible at all to get rid of the container?
>>
>> Not without making some code changes.
>
>> all what the container would do.
> Yeah, that was logical. But I mean the JAMES code. Are there any changes
> that must be done in the JAMES code to achieve that? How much effort
> would it be?
>
> Also, were you able to totally get rid of the container jars+code?
>
> Thanks in advance,
Keep in mind that get rid of the container (Phoenix) is a different
thing than get rid of the Avalon framework.
Btw I think that the avalon framework itself is a good thing to james
and we could adapt james to run under j2ee or spring without getting rid
of avalon interfaces.
I think most threads here to remove the framework code is "fear of the
unknown". I also was against avalon before I studied it and used it
working on James. If we want to remove the avalon interfaces we should
copy the interfaces to our codebase... what do we get but not being
avalon compatible?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Ahmed Mohombe <am...@yahoo.com>.
> So I guess with keeping the avalon framework no changes to
> James would be necessary, but this was not my situation.
Neither mine :). If one would want to get rid of the container than ... :).
>> How much effort would it be?
>
> It depends on the knowledge of phoenix and james. The container
> replacement is not much code to write (~150 lines), but the
> configuration and the server (socket creation) was something I rewrote,
> because I figured adapting would be more effort.
>
> Assuming all uncertainties turn out positive, less than a week.
Thank you very much for your feedback,
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Ahmed Mohombe" <am...@yahoo.com>
To: <se...@james.apache.org>
Sent: Monday, March 06, 2006 4:30 PM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
> Are there any changes that must be done in the JAMES code to achieve that?
I refactored the avalon framework and james, but that would not be
necessary. So I guess with keeping the avalon framework no changes to James
would be necessary, but this was not my situation.
> How much effort would it be?
It depends on the knowledge of phoenix and james. The container replacement
is not much code to write (~150 lines), but the configuration and the server
(socket creation) was something I rewrote, because I figured adapting would
be more effort.
Assuming all uncertainties turn out positive, less than a week.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Ahmed Mohombe <am...@yahoo.com>.
>>> I don't use a container like phoenix.
>> Is this possible? Is it possible at all to get rid of the container?
>
> Not without making some code changes.
> all what
> the container would do.
Yeah, that was logical. But I mean the JAMES code. Are there any changes that must be done in the
JAMES code to achieve that? How much effort would it be?
Also, were you able to totally get rid of the container jars+code?
Thanks in advance,
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Ahmed Mohombe" <am...@yahoo.com>
To: <se...@james.apache.org>
Sent: Monday, March 06, 2006 3:58 PM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
>> I don't use a container like phoenix.
> Is this possible? Is it possible at all to get rid of the container?
Not without making some code changes. I have a class that reads the config
files, instantiates the loggers and starts the servers, all what the
container would do.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Ahmed Mohombe <am...@yahoo.com>.
> I don't use a container like phoenix.
Is this possible? Is it possible at all to get rid of the container?
I also see no reason to use one as it doesn't deliver the most important thing I would expect
from a container: "reloading the configuration without restarting the server".
Ahmed.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Stefano Bagnara" <ap...@bago.org>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, March 06, 2006 2:40 PM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
> "Different" james?
> I'm curious about the differences and what kind of modularity would be
> needed in james so that it would be the same version but using additional
> plugins/module/components.
I don't use a container like phoenix.
I have two spooler; one for incoming messages and another one for messages
that produced an error.
The first one operates only on the tmp-file, so no db-operations.
>From the server classes I only use the handler.
Error handling in the Linearprocessors is different.
JDBC mailstore classes can write the body of the message into the file rep,
as discussed before.
This also results in different MimeMessageWrapper and the like classes.
My version is designed for speed, simplicity and extensive error handling.
Also I should be able to fix every problem in a reasonable time, so a
dependency like phoenix was not an option.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Markus Kühn wrote:
>> Btw, we're not removing dbfiles or files repositories, I'm just trying
>> to make db repositories to work fine with big files (when the
>> underlying db correctly support streams) and this can be achieved only
>> via jdbc streams. You can keep using dbfiles, as you already did (I
>> also improved this scenario by altering the default "file" stream
>> repositories to provide sharedinputstreams).
>
> No worries, I use a different james version. And that's the reason why
> my changes are of no use to the james project.
"Different" james?
I'm curious about the differences and what kind of modularity would be
needed in james so that it would be the same version but using
additional plugins/module/components.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Stefano Bagnara" <ap...@bago.org>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, March 06, 2006 2:00 PM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
> Btw, we're not removing dbfiles or files repositories, I'm just trying to
> make db repositories to work fine with big files (when the underlying db
> correctly support streams) and this can be achieved only via jdbc streams.
> You can keep using dbfiles, as you already did (I also improved this
> scenario by altering the default "file" stream repositories to provide
> sharedinputstreams).
No worries, I use a different james version. And that's the reason why my
changes are of no use to the james project.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Markus Kühn wrote:
> I think the biggest problem with dababase streams is, that they do work
> only with some databases.
I agree, but you'll find the same problems with file greater than 2GB:
they are not supported by all the filesystems ;-) Other filesystems will
limit the maximun number of files per directory, and so on...
Btw, we're not removing dbfiles or files repositories, I'm just trying
to make db repositories to work fine with big files (when the underlying
db correctly support streams) and this can be achieved only via jdbc
streams. You can keep using dbfiles, as you already did (I also improved
this scenario by altering the default "file" stream repositories to
provide sharedinputstreams).
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Stefano Bagnara" <ap...@bago.org>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, March 06, 2006 11:54 AM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
> Markus Kühn wrote:
>> I didn't and wouldn't change MimeMessageJDBCSource with all the problems
>> and side effects you described. I found storing the body of big messages
>> by default in the file system to be the best solution for me so far.
>
> Well, I use db repositories because I need a good failover solution and
> simpler backups. Most db provide replication, also.
>
> When all the data is in a single place it's much more easy to manage it
> and avoid "synchronization" problems between filesystem and db data.
Enabling replication or fail over for files shouldn't be hard.
Administrators just have to be aware of the situation, that they could
activate or deactivate an OutofMemory-safety by using dbfile for big
messages.
> The changes to MimeMessageJDBCSource are needed for DB repositories while
> your solution is good for messages created from scratch.
>From scratch and also existing messages on the file system, as you pointed
out below.
> Using dbfile or file repositories we can just use the SharedInputStream
> from the stream repository (patch pending).
I think the biggest problem with dababase streams is, that they do work only
with some databases.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Markus Kühn wrote:
> I didn't and wouldn't change MimeMessageJDBCSource with all the problems
> and side effects you described. I found storing the body of big messages
> by default in the file system to be the best solution for me so far.
Well, I use db repositories because I need a good failover solution and
simpler backups. Most db provide replication, also.
When all the data is in a single place it's much more easy to manage it
and avoid "synchronization" problems between filesystem and db data.
The changes to MimeMessageJDBCSource are needed for DB repositories
while your solution is good for messages created from scratch. Using
dbfile or file repositories we can just use the SharedInputStream from
the stream repository (patch pending).
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Markus Kühn <ty...@hotmail.com>.
----- Original Message -----
From: "Stefano Bagnara" <ap...@bago.org>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, March 06, 2006 12:52 AM
Subject: Re: MimeMessage handling optimizations/changes for 2.3.0a2
> I changed the MimeMessageJDBCSource to provide a SharedInputStream from
> the db, but I'm not sure this is a good idea:
I didn't and wouldn't change MimeMessageJDBCSource with all the problems and
side effects you described. I found storing the body of big messages by
default in the file system to be the best solution for me so far. The good
thing is that it is fast (only move the tmp-file), no changes to
MimeMessageJDBCSource are needed and it won't happen very often. I currently
use a size-threshold of 5 MB. The important thing for me is to prevent the
big ones from being loaded and not every message.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by Stefano Bagnara <ap...@bago.org>.
Noel J. Bergman wrote:
>> I though that I should have tried to drop the "wrapping/delegation"
>> behaviour in favor of a simple extension.
>
> I recall thinking the same. And moving to a shared inputstream and
> streaming should be a big win on footprint. Dan also mentioned it, with
> respect to Derby.
I just reread a thread (10-25 January 2006) with comments and code by
Markus Kuhn and Ofir Gross and maybe we should also incorporate some of
their suggestions. E.g: we could provide more constructors to
MimeMessageWrapper so that we can use it instead of MimeMessage when
"cloning" other MimeMessages.
Then we could simply fallback to a temporary sharedfile for the
contentStream as like as we do while receiving smtp messages with
MimeMessageInputStreamSource (that I already converted to provide
sharedinputstreams, in my test).
>> This way it was simpler to define when the message is needed and I've
>> been able to avoid the message loading for most header operations
>> (previously needed).
>
> This is a large part of what the wrapper was originally written to do, IIRC.
> Way before I started with JAMES.
Yes, but previously the full message (instead of headers) was loaded for
each setHeader operation. Now also set operations on headers try to
avoid the message loading.
>> I made [streaming] optional by adding a new "getBody" option
>> (useBinaryStream) to the dbOptions in sqlResources so that
>> testers could try this feature over different dbs and provide
>> feedback.
>
> If you can complete the changes, I'd be in favor of getting them into the
> next milestone.
I've done more tests with db repositories (not dbfile) over mysql, derby
and oracle and big messages (100MB+) and limited Xmx (40MB for mysql
and derby, 50MB for oracle and hsqldb) and I found something:
I need to use a PipedInputStream/PipedOutputstreams with an additional
buffering thread in order to store the message using streams for both
mysql and derby. I tested another option (setBinaryStream on a blob
obtained with "select for update") where this was not needed but both db
throws various "not implemented" exceptions.
IMHO this is not a showstopper because Activation framework (JAF)
already spawn similar threads when working with mimemessages. The main
problem is that the exception handling is much more difficult with this
kind of "worker" threads around. On the performance side we probably
want to run the thread when the message is larger of, as an example,
10KB. With smaller messages we can use a bytearray buffer and avoid the
additional thread (I prefer to add this easy optimization after we
tested to correctness of the streaming methods).
On the mysql side the streaming operations works correctly only using
mysql 4.1+, connector/j 5+ (latest connector/j 3.1.12 still have bugs)
and adding an "emulateLocators=true" parameter in the jdbc url.
The bad news is that using the streaming methods with older
connectors/servers ends in corrupted/truncated data!
Testing hsqldb I get OutOfMemoryExceptions and Corrupted Data all the
times so I skipped it.
The great news is that the same identical queries works with Oracle too!
I still can't fully believe this thing (I work often with jdbc/oracle
and I always had to write specific code). I only tested oracle 9.2.0.4
accessed by the latest 10G thin jdbc driver (10.1.0.4.0) (older drivers
didn't work).
I changed the MimeMessageJDBCSource to provide a SharedInputStream from
the db, but I'm not sure this is a good idea: currently if a message
created from that source is parsed javamail open a log of new shared
streams (creates new connection/queries to db). Maybe I should try to
intercept the parsing and copy the stream to a temporary shared file
instead of opening new connections (not totally sure it would be better)
Now I need a few (many?!) hours to put order on what I wrote and I hope
I'll commit something soon. After the commit and a few more tests we'll
vote on what to do with defaults. We probably should defaults to
binarystreams for derby, as we exactly know what version of derby is
used, and use standard bytearray operations for oracle and mysql and
perhaps add a note in the config.xml explaining how to enable stream
operations and what driver version/server version is needed.
As a side note I've seen that other players (activemq and jboss mail
server) are using another approach to save big messages: they split the
message over multiple records. I personally don't like this solution and
I prefer a big restriction on the usable jdbc drivers but a full
streaming support or as a last resort the file (or dbfile) repository.
Stefano
PS: I would like to alter the SMTPServer to directly stream incoming
messages to DB, but unfortunately I found out that we have to know the
exact size of the message before being able to stream it to db!
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: MimeMessage handling optimizations/changes for 2.3.0a2
Posted by "Noel J. Bergman" <no...@devtech.com>.
> I though that I should have tried to drop the "wrapping/delegation"
> behaviour in favor of a simple extension.
I recall thinking the same. And moving to a shared inputstream and
streaming should be a big win on footprint. Dan also mentioned it, with
respect to Derby.
> This way it was simpler to define when the message is needed and I've
> been able to avoid the message loading for most header operations
> (previously needed).
This is a large part of what the wrapper was originally written to do, IIRC.
Way before I started with JAMES.
> The result is that all the tests still passes and that I removed a full
> layer.
> I made [streaming] optional by adding a new "getBody" option
> (useBinaryStream) to the dbOptions in sqlResources so that
> testers could try this feature over different dbs and provide
> feedback.
If you can complete the changes, I'd be in favor of getting them into the
next milestone.
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org