You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "motet.dk, cl" <cl...@motet.dk> on 2005/06/20 14:09:38 UTC
Re: [jira] Commented: (JAMES-134) Large emails in the spool cause SpoolManager to throw OutOfMemoryError
Hello,
Cannot sending large emails is a very big issue for an enterprise mail server.
We can hope an enterprise server is able to send at least a 10 Mo email.
James is able to do that, but the performance cost is very expensive :
- important memory increasing
- important CPU increasing
- important slow down of the server performance
I read the dicussion about this subject at :
http://issues.apache.org/jira/browse/JAMES-134?page=comments#action_63254
If i have well understood it, the causes are :
- the management of the memory by the JVM
- the using of the JavaMail API
I have some question about this topic :
- This discussion happens two months ago, have you found a solution to this issue since ?
- Has someone found a way to improve the actual behavior ? some ideas ?
- You speak about 'new JavaMail content handlers', has someone tried to implement a such handler ?
- Has someone tried to use another JVM than the SUN JVM ?
Our company really want to use James in solutions for our clients, because it's a very good product,
but this issue could cause we change our mind if it's not solved.
If we need to change the source code and use another opensource project for replace the JavaMail it wouldn't be a problem, we simply need to know if this issue have a solution, and if the solution can be implemented fastly.
Thanks all for your answers.
Best regards,
Christophe
------------------------------------------------------
This mail was sent through Jay.net: http://www.jay.net
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [jira] Commented: (JAMES-134) Large emails in the spool cause SpoolManager to throw OutOfMemoryError
Posted by Mark Daring <ty...@hotmail.com>.
>> There's no need to create a MimeMessage.
>
> You do if you want to make any changes to the message, e.g., add a footer,
> and until we integrate MIME4J, you need a MimeMessage anytime you want to
> parse. Generally, we use our own MimeMessageWrapper, which doesn't load
> the
> message unless necessary.
In this special case there's no way around loading the message into memory,
unless someone
uses logic like in e.g. MimeMultiPart.parse() for multipart messages without
a ByteArrayStream
as buffer. But most often no changes must be applied and so the MimeMessage
doesnt need to
created.
>> if you add a custom header you dont have to load the entire
>> message into memory by calling saveChanges() like in the
>> LocalDelivery-Mailet.
>
> If I recall correctly, that statement is demonstrably false, and was
> tested
> to make sure.
I don't know who tested what how but it works for me.
Just take a look at MimeMessage.saveChanges() and
MimeMessage.writeTo(OutputStream os, String[] ignoreList).
All additional headers are stored in a separate list, which is written into
the
outputstream the time writeto is invoked.
LocalDelivery looks like this:
public void service(Mail mail) throws MessagingException {
..
// Prevent MimeMessage loading into memory
if (message instanceof MimeMessageWrapper)
((MimeMessageWrapper)message).addWrapperHeader("Delivered-To",
recipient.toString());
..
and MimeMessageWrapper:
..
/**
* Adds a Header to the wrapper only. The original MimeMessage is not
affected.
* .writeTo(OutputStream headerOs, OutputStream bodyOs, String[]
ignoreList) will write this
* header into the outputstream.
* Purpose: Prevent loading the MimeMessage into memory.
* @param name
* @param value
* @throws MessagingException
*/
public void addWrapperHeader(String name, String value) throws
MessagingException {
if (wheaders == null) {
wheaders = new MailHeaders();
}
wheaders.addHeader(name, value);
}
..
>> MySQL has (had) a problem with BLOBS and streams;
>
>> rewritting the spool and MailImpl to use the FileInputStream
>> from the tmp-file the e-mail was stored in, can solve this
>> problem, especially when all e-mails of a certain size are
>> moved automatically into a FileStore even if a database as
>> MailStore was specified.
>
> That wouldn't when a database is used for clustering ...
Actually the point is that the FileInputStream of the tmp-file should be
used
wherever possible. This way my database never gets in touch with the content
of
the e-mail serviced by RemoteDelivery cause it writes directly from the
FileInputStream.
And if RemoteDelivery fails I only save the path of the file.
If someone wants use a cluster I'd say that by default this behaviour should
be
disabled, unless the software takes care of it.
MD
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [jira] Commented: (JAMES-134) Large emails in the spool cause SpoolManager to throw OutOfMemoryError
Posted by Mark Daring <ty...@hotmail.com>.
As I have had a short look at the JavaMail source code, maybe if the
inputstream
used for MimeMessage implements SharedInputStream this could also prevent
loading the entire message into memory. At least for MimeMultiPart it doesnt
look bad,
but I have not tried, cause I dont create the MimeMessage at all (and also
dont alter it).
MD
----- Original Message -----
From: "Noel J. Bergman" <no...@devtech.com>
To: "James Developers List" <se...@james.apache.org>
Sent: Monday, June 20, 2005 9:59 PM
Subject: RE: [jira] Commented: (JAMES-134) Large emails in the spool cause
SpoolManager to throw OutOfMemoryError
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
RE: [jira] Commented: (JAMES-134) Large emails in the spool cause SpoolManager to throw OutOfMemoryError
Posted by "Noel J. Bergman" <no...@devtech.com>.
Mark Daring wrote:
> I think the problem is that JAMES doesnt use the ways around this problem.
Patches welcomed.
> There's no need to create a MimeMessage.
You do if you want to make any changes to the message, e.g., add a footer,
and until we integrate MIME4J, you need a MimeMessage anytime you want to
parse. Generally, we use our own MimeMessageWrapper, which doesn't load the
message unless necessary.
> if you add a custom header you dont have to load the entire
> message into memory by calling saveChanges() like in the
> LocalDelivery-Mailet.
If I recall correctly, that statement is demonstrably false, and was tested
to make sure.
> MySQL has (had) a problem with BLOBS and streams;
> rewritting the spool and MailImpl to use the FileInputStream
> from the tmp-file the e-mail was stored in, can solve this
> problem, especially when all e-mails of a certain size are
> moved automatically into a FileStore even if a database as
> MailStore was specified.
That wouldn't when a database is used for clustering ...
--- Noel
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [jira] Commented: (JAMES-134) Large emails in the spool cause SpoolManager to throw OutOfMemoryError
Posted by Mark Daring <ty...@hotmail.com>.
> The "problem" is probably not the Javamail APIs but the SUN Javamail
> implementation.
I think the problem is that JAMES doesnt use the ways around this problem.
Richs comment
(http://issues.apache.org/jira/browse/JAMES-134?page=comments#action_63254)
was absolutly right. There's no need to create a MimeMessage. And even if
you add a custom header
you dont have to load the entire message into memory by calling
saveChanges() like in the LocalDelivery-Mailet.
Another thing is that at MySQL has (had) a problem with BLOBS and streams;
this means
on loading the e-mail from the db the whole thing is in memory. So
rewritting the spool and MailImpl
to use the FileInputStream from the tmp-file the e-mail was stored in, can
solve this problem, especially
when all e-mails of a certain size are moved automatically into a FileStore
even if a database as MailStore
was specified.
Though this ideas are not new, maybe now is the best time to implement them.
MD
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: [jira] Commented: (JAMES-134) Large emails in the spool cause SpoolManager to throw OutOfMemoryError
Posted by Stefano Bagnara <ap...@bago.org>.
> Cannot sending large emails is a very big issue for an
> enterprise mail server.
> We can hope an enterprise server is able to send at least a
> 10 Mo email.
>
> James is able to do that, but the performance cost is very expensive :
> - important memory increasing
> - important CPU increasing
> - important slow down of the server performance
I currently use James and it handles 10-20MB mails with no problems or
"important CPU usage".
I could be optimized but it works.
> If i have well understood it, the causes are :
> - the management of the memory by the JVM
I think the JVM handle the memory correctly for modern operating systems.
> - the using of the JavaMail API
The "problem" is probably not the Javamail APIs but the SUN Javamail
implementation.
> I have some question about this topic :
> - This discussion happens two months ago, have you found a
> solution to this issue since ?
> - Has someone found a way to improve the actual behavior ?
> some ideas ?
> - You speak about 'new JavaMail content handlers', has
> someone tried to implement a such handler ?
> - Has someone tried to use another JVM than the SUN JVM ?
I think that the problem is not the JVM.
> Our company really want to use James in solutions for our
> clients, because it's a very good product, but this issue
> could cause we change our mind if it's not solved.
Have you tested James? Try using it, send a large message and look at the
"performance": my deployed James succesfully handle 10-20Mb messages.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org