You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Edward Tan <ta...@gmail.com> on 2005/12/30 12:39:23 UTC
Fatal Exception when receiving email > 30 MB
Hi,
James suddenly crash when I receive > 30MB attachment. I didn't check the
threshold of message size that causes this. I just happened to send 34 MB
attachment through Thunderbird to James. The problem is caused by my mailet
which tries to get the flag:
in my my mailet:
message = mail.getMessage();
.....
// FLAGS
---> (exception when calling this:) Flags flags =
message.getFlags();
The exception is caught in org.apache.james.transport.JamesSpoolManager
line 233:
} catch (Throwable e) {
// This is a strange error situation that shouldn't
ordinarily
// happen
StringBuffer exceptionBuffer =
new StringBuffer(64)
.append("Exception in processor <")
.append(processorName)
.append(">");
getLogger().error(exceptionBuffer.toString(), e);
if (processorName.equals(Mail.ERROR)) {
// We got an error on the error processor...
// kill the message
mail.setState(Mail.GHOST);
mail.setErrorMessage(e.getMessage());
Has anyone experienced before?
Any limit on message size?
Regards,
Edward
Re: Weird behaviour with remote db repository
Posted by Stefano Bagnara <ap...@bago.org>.
Carl Vorster wrote:
> Thanks Stefano for the reply, but in my debugging process I created a new
> empty db and still got the same results - any other thoughts ?
Maybe the problem is in your network or in name resolution between the 2
machines.
How much does it take to run a mysql client from the same machine where
you installed james to the same machine where you have the james db?
Please use the same username/port/password/host as in your jdbc
configuration and test a simple query: it should execute in 0.1secs or
less... if it take more then the problem is in your network.
If it stopped working correctly suddenly then it is probably a dns or an
ethernet cable problem.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
RE: Weird behaviour with remote db repository
Posted by Carl Vorster <cc...@propartners.co.za>.
Thanks Stefano for the reply, but in my debugging process I created a new
empty db and still got the same results - any other thoughts ?
Thanks
Carl
-----Original Message-----
From: Stefano Bagnara [mailto:apache@bago.org]
Sent: Friday, December 30, 2005 5:51 PM
To: James Users List
Subject: Re: Weird behaviour with remote db repository
Carl Vorster wrote:
> Have any of you ran into a similar situation, or can point me into the
right
> direction of why this is happening?
> mysql <yourdb>
repair table spool;
repair table inbox;
exit;
Let me know if this solve your problem.
Mysql tables must be manually optimized (repair run the optimization
too) when a lot of write/delete happens.
If this fix the problem you probably want to put a cron script to run
the repair.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
--
This message has been scanned for viruses and
dangerous content by Modiredi Internet Services(http://www.modiredi.co.za).
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Weird behaviour with remote db repository
Posted by Stefano Bagnara <ap...@bago.org>.
Carl Vorster wrote:
> Have any of you ran into a similar situation, or can point me into the right
> direction of why this is happening?
> mysql <yourdb>
repair table spool;
repair table inbox;
exit;
Let me know if this solve your problem.
Mysql tables must be manually optimized (repair run the optimization
too) when a lot of write/delete happens.
If this fix the problem you probably want to put a cron script to run
the repair.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Weird behaviour with remote db repository
Posted by Carl Vorster <cc...@propartners.co.za>.
Hi,
Mail server:
James 2.2.0
JDK 1.5
Windows 2000 server
Repository:
MySql 4.1 on dedicated db server
Windows 2000 server
I've been running James 2.2.0 in a production environment without problems
for a couple of months now. Yesterday James's response times suddenly
dropped to almost standstill.
SMTP receives new connections, but a lot of client connections times out.
The root processor processes the spool; but very slowly i.e. 1 mail every 2
minutes.
Remote delivery does the same.
There's nothing in the log files except the normal stuff.
I debugged to a point where I can simulate the behavior and found that it's
because the db is sitting on another server.
If I run with a file store everything is fine.
If I run with a db store on the same server everything is fine.
I then looked at the dedicated db server but again couldn't find anything
out of place; the server also hosts other db's for other applications and
there's no degradation in service on the other apps. The server only runs at
5% capacity.
Have any of you ran into a similar situation, or can point me into the right
direction of why this is happening?
Thanks in advance
Carl
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
Flags flags = message.getFlags();
And I tried java mail 1.4ea and it is the same as 1.3.2, it runs out of
heap.
I think sun Java implementation to put everything into memory is not
scalable. We are talking about email attachment and not java POJO. And for
very busy server like one used in ISP, this is not scalable.
Anyway, good work. James 2.2.0 is very stable so far.
Regards,
Edward
On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> > Is it possible to store it into temporary file? Using more memory does
> not
> > guarantee the stability, unless the email size is limited. But this is
> not
> > practical since it reduces James scalability.
>
>
> There's no way we can control how sun javamail handle our messages.
> We should write our own javamail implementation and this is a big work.
>
> We already wrapped the MimeMessage to avoid loading the message when
> possible but sometimes javamail will load the message anyway!
>
> What is the mimemessage operation you are calling when you see the
> OutOfMemoryError?
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
> Is it possible to store it into temporary file? Using more memory does not
> guarantee the stability, unless the email size is limited. But this is not
> practical since it reduces James scalability.
There's no way we can control how sun javamail handle our messages.
We should write our own javamail implementation and this is a big work.
We already wrapped the MimeMessage to avoid loading the message when
possible but sometimes javamail will load the message anyway!
What is the mimemessage operation you are calling when you see the
OutOfMemoryError?
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
Hi Stefano,
I have tried
ByteArrayInputStream headersIn = new
SharedByteArrayInputStream(headers.toByteArray());
but it still runs out of heap.
Is it possible to store it into temporary file? Using more memory does not
guarantee the stability, unless the email size is limited. But this is not
practical since it reduces James scalability.
Thanks.
Regards,
Edward
On 12/30/05, Edward Tan <ta...@gmail.com> wrote:
>
> Yes!
>
> } catch(Throwable t) {
> System.out.println(t.getMessage());
> t.printStackTrace();
>
> ====>
>
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
>
>
> On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
> >
> > Edward Tan wrote:
> > > The exception is caught somewhere, so I set at earlier calling
> > function
> > > stack.
> > > But
> > >
> > > message is null
> > >
> > > and it goes to finally.
> >
> > If it execute the finally, the message is null and the 2
> > System.out.printlm are not called then it could be a throwable (probably
> > an error, an out of memory error?)
> >
> > If you can reproduce you could add a
> > } catch (Throwable t) {
> > // some output code
> > throw t;
> > } finally ...
> >
> > Stefano
> >
> > > Somewhere the exception is caught and never thrown back.
> > >
> > > ------------------
> > > private synchronized void loadMessage() throws MessagingException
> > {
> > > if (message != null) {
> > > //Another thread has already loaded this message
> > > return;
> > > }
> > > InputStream in = null;
> > > try {
> > > in = source.getInputStream ();
> > > headers = new MailHeaders(in);
> > >
> > > ByteArrayInputStream headersIn
> > > = new ByteArrayInputStream(headers.toByteArray());
> > > in = new SequenceInputStream(headersIn, in);
> > >
> > > message = new MimeMessage(session, in);
> > > } catch (IOException ioe) {
> > > System.out.println(ioe);
> > > throw new MessagingException("Unable to parse stream: " +
> > > ioe.getMessage(), ioe);
> > > } catch (Exception e) {
> > > System.out.println(e);
> > > e.printStackTrace ();
> > > } finally {
> > > IOUtil.shutdownStream(in);
> > > }
> > > }
> > > ----------------------------
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
> >
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
Hi,
Thanks. Replacing javamail with mime4j is a massive works. Actually, I am
only interested in making sure that large emails will be successfully
delivered. Increasing java VM memory through command line is one way. But
with limited RAM (though I can put 1GB - 2GB RAM) won't guarantee that any
email will not dissapear.
I just found out that when I disabled all mailet except transport mailet
LocalDelivery, large email still crash (30+ MB) with default parameters for
java VM. Though at 13+ MB, the email is still delivered. I am not sure if
anyone using James out of the box experiencing mail error when handling
large attachment.
Is there anyway that to avoid the heap out of memory? Maybe something like
swapping to file storage. Maybe java VM has parameter to set that it uses
file storage as swapping when it requires more memory. This will solve my
problem though at the expense of performance.
Regards,
Edward
On 12/31/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> > Has James team considered using mime4j as a drop-in replacement to
> javamail?
> > I googled around and noticed that someone has incorporated mime4j in
> James.
>
> Mime4j is a readonly API.
> To fully replace Javamail we would need a read/write API.
>
> The options are 2:
>
> - use a different javamail implementation: Starting from geronimo mail
> implementation, but geronimo javamail is still young and doesn't support
> most of sun javamail features.
>
> - use a non "javax.mail" mail handling API (And I don't know any good
> ASL library to manipulate mime messages)
>
> This is a big work anyway and no james committer has enough time
> currently.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Getting a "java.net.ConnectException: Connection timed out: connect"
now
Posted by Bernd Fondermann <bf...@brainlounge.de>.
david joffrin wrote:
> Hi everyone,
>
> Hope you can help as I am getting into troubles now.
>
> I have changed my ADSL provider (UK). Everything was fine before, now,
> for each mail (including mail to hotmail), I am getting a
> "java.net.ConnectException: Connection timed out: connect".
>
> Is there anything I have missed? Is there any limitation with JAMES that
> I am not aware depending on the ADSL connection I might have.
as long as your DSL provider provides you with TCP/IP, there should be
no limitations ;-)
sounds more like a routing/firewall issue.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Getting a "java.net.ConnectException: Connection timed out: connect" now
Posted by david joffrin <da...@hotmail.com>.
Hi everyone,
Hope you can help as I am getting into troubles now.
I have changed my ADSL provider (UK). Everything was fine before, now, for
each mail (including mail to hotmail), I am getting a
"java.net.ConnectException: Connection timed out: connect".
Is there anything I have missed? Is there any limitation with JAMES that I
am not aware depending on the ADSL connection I might have.
Thanks.
DvJ
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
> Has James team considered using mime4j as a drop-in replacement to javamail?
> I googled around and noticed that someone has incorporated mime4j in James.
Mime4j is a readonly API.
To fully replace Javamail we would need a read/write API.
The options are 2:
- use a different javamail implementation: Starting from geronimo mail
implementation, but geronimo javamail is still young and doesn't support
most of sun javamail features.
- use a non "javax.mail" mail handling API (And I don't know any good
ASL library to manipulate mime messages)
This is a big work anyway and no james committer has enough time currently.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
I can understand why we need to use javamail since I have found the
saveChanges at almost all mailets in James.
org/apache/james/transport/mailets/AddHabeasWarrantMark.java:
message.saveChanges();
org/apache/james/transport/mailets/LocalDelivery.java:
localMessage.saveChanges();
org/apache/james/transport/mailets/AddFooter.java:
message.saveChanges();
org/apache/james/transport/mailets/AddHeader.java:
message.saveChanges();
org/apache/james/transport/mailets/DSNBounce.java: newMail.getMessage
().saveChanges();
org/apache/james/transport/mailets/DSNBounce.java:
dsnMessage.saveChanges();
org/apache/james/transport/mailets/AbstractRedirect.java:
newMail.getMessage().saveChanges();
org/apache/james/transport/mailets/ServerTime.java:
response.saveChanges();
org/apache/james/James.java: reply.saveChanges();
org/apache/james/core/MimeMessageWrapper.java:
message.saveChanges();
org/apache/james/core/MimeMessageWrapper.java: public void saveChanges()
throws MessagingException {
org/apache/james/core/MimeMessageWrapper.java: message.saveChanges();
org/apache/james/fetchpop/FetchPOP.java:
message.saveChanges();
org/apache/james/fetchmail/MessageProcessor.java:
messageOut.saveChanges();
On my first rough guess, changing mime4j to my problem of receiving large
email will not be so easy. I can change my mailet to use mime4j but other
James' mailets are still using javamail and replacing them will be masive
work since most of them requires writing to the message.
If I narrow down to my requirement, the work will be concentrated in
LocalDelivery mailet but still this is not a pretty solution.
Has James team considered using mime4j as a drop-in replacement to javamail?
I googled around and noticed that someone has incorporated mime4j in James.
Regards,
Edward
On 12/31/05, Edward Tan <ta...@gmail.com> wrote:
>
> And for the same 30+ MB message.
>
> It is about 8 seconds. (I tried several times and it is the same about 8
> seconds and uses without extra memory, default java).
>
> and for 190+ MB file, it takes 1 minute 32 seconds.
>
> I cannot think how it is different.
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
Sorry,
I make amendments. I did the test using the mime4j java app using Swing GUI.
But just now I used my mailet:
public void mime4jService(Mail mail) throws MessagingException {
MimeMessageWrapper mimeMessageWrapper = (MimeMessageWrapper)
mail.getMessage();
try {
InputStream is = mimeMessageWrapper.source.getInputStream();
org.mime4j.message.Message message = new
org.mime4j.message.Message(is);
} catch (IOException ioe) {
System.out.println(ioe);
ioe.printStackTrace();
}
}
public void service(Mail mail) throws MessagingException {
boolean bUseMime4j = true;
mime4jService(mail);
if (bUseMime4j) return;
.....
}
And for the same 30+ MB message.
It is about 8 seconds. (I tried several times and it is the same about 8
seconds and uses without extra memory, default java).
and for 190+ MB file, it takes 1 minute 32 seconds.
I cannot think how it is different.
And the FileStreamStore and FileObjectStore goes to error folder when I use
the above codes. Do I need to set the
mail.setState(Mail.TRANSPORT);
like:
public void mime4jService(Mail mail) throws MessagingException {
MimeMessageWrapper mimeMessageWrapper = (MimeMessageWrapper)
mail.getMessage();
try {
InputStream is = mimeMessageWrapper.source.getInputStream();
org.mime4j.message.Message message = new
org.mime4j.message.Message(is);
mail.setState(Mail.TRANSPORT);
} catch (IOException ioe) {
System.out.println(ioe);
ioe.printStackTrace();
}
}
Regards,
Edward
On 12/31/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> Edward Tan wrote:
> > I have tried mime4j just now and I compared it with James that uses
> Javax
> > mail.
>
> Thank you for your tests!
>
> > So it seems a bit dangerous to use javax.mail since we will never know
> when
> > some emails will go to error folder because of OutOfMemory.
>
> Mime4j let us only to read messages and not to write/modify them. So we
> are stick to javamail by now. Furthermore we exposed javamail
> MimeMessage in Mailet APIs so we will able to change this dependency in
> james 3.0+.
>
> This tests are indeed useful to start planning changes for the next
> major release (3.0).
>
> > ----------------
> >
> > Other problem I encountered using mime4j (sample tester: MessageTree),
> it
> > raises exception when processing text/plain with charset ISO-8859-1.
> >
> > I saw that someone has incorporated mime4j into James, though still
> > experimental. But I don't find any version of James that contains
> mime4j.
>
> Sorry, but I've got no answers for this.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
Edward Tan wrote:
> I have tried mime4j just now and I compared it with James that uses Javax
> mail.
Thank you for your tests!
> So it seems a bit dangerous to use javax.mail since we will never know when
> some emails will go to error folder because of OutOfMemory.
Mime4j let us only to read messages and not to write/modify them. So we
are stick to javamail by now. Furthermore we exposed javamail
MimeMessage in Mailet APIs so we will able to change this dependency in
james 3.0+.
This tests are indeed useful to start planning changes for the next
major release (3.0).
> ----------------
>
> Other problem I encountered using mime4j (sample tester: MessageTree), it
> raises exception when processing text/plain with charset ISO-8859-1.
>
> I saw that someone has incorporated mime4j into James, though still
> experimental. But I don't find any version of James that contains mime4j.
Sorry, but I've got no answers for this.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
I have tried mime4j just now and I compared it with James that uses Javax
mail.
For the same 30+ MB message:
Timing:
mime4j: around 1 minute 20 seconds
javax.mail: 10 seconds
Memory:
mime4j -> without extra memory (default java -without any argument)
javax.mail -> -Xms512M -Xmx768M (without this extra, it fails)
For the 200+ MB file:
Timing:
mime4j: 7 minute and 10 seconds
javax.mail: fails even with extra memory -Xms512M -Xmx768M
Memory:
mime4j: the same as with 30MB
Other observations:
Mime4j uses nice processes
javax.mail uses user processes
Mime4j when running, my mouse is still responsive
javax.mail, my mouse is not really responsive
So it seems a bit dangerous to use javax.mail since we will never know when
some emails will go to error folder because of OutOfMemory.
----------------
Other problem I encountered using mime4j (sample tester: MessageTree), it
raises exception when processing text/plain with charset ISO-8859-1.
I saw that someone has incorporated mime4j into James, though still
experimental. But I don't find any version of James that contains mime4j.
Thanks.
Regards,
Edward
On 12/31/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> > getFlags is not really important. But it also crashes when I try to get
> the
> > filename of the attachment, i.e. message.getFilename (of the Part). I am
> not
> > sure if there is other way to get the filename of the attachments. The
> email
> > header certainly does not contain the filename of the attachment.
>
>
> I think there is no easy solution to your problem.
>
> The only idea I have is the following:
>
> - Check wether the MimeMessage you get is a MimeMessageWrapper (it
> should if it's read from the repository), if it's a MimeMessageWrapper
> then change the MimeMessageWrapper code to give you visibility of the
> source "MimeMessageSource source = null;" (now it has package visibility)
>
> - Parse the message source with a different mime library (mime4j:
> http://mime4j.sourceforge.net/, look for the MimeStreamParser class).
> Mime4J should be more memory friendly than javamail.
>
> I'm not sure this will work or even fix the memory problem, but this is
> the first attempt I would do.
>
> Let us know of your progress!
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
> getFlags is not really important. But it also crashes when I try to get the
> filename of the attachment, i.e. message.getFilename (of the Part). I am not
> sure if there is other way to get the filename of the attachments. The email
> header certainly does not contain the filename of the attachment.
I think there is no easy solution to your problem.
The only idea I have is the following:
- Check wether the MimeMessage you get is a MimeMessageWrapper (it
should if it's read from the repository), if it's a MimeMessageWrapper
then change the MimeMessageWrapper code to give you visibility of the
source "MimeMessageSource source = null;" (now it has package visibility)
- Parse the message source with a different mime library (mime4j:
http://mime4j.sourceforge.net/, look for the MimeStreamParser class).
Mime4J should be more memory friendly than javamail.
I'm not sure this will work or even fix the memory problem, but this is
the first attempt I would do.
Let us know of your progress!
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
Hi,
getFlags is not really important. But it also crashes when I try to get the
filename of the attachment, i.e. message.getFilename (of the Part). I am not
sure if there is other way to get the filename of the attachments. The email
header certainly does not contain the filename of the attachment.
I need this because I need to extract attachment of the email to store in
shared central repository based on certain criteria of the email. I am doing
project for my client with this kind of requirement.
Thanks.
Regards,
Edward
On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> > Is it possible to store it into temporary file? Using more memory does
> not
> > guarantee the stability, unless the email size is limited. But this is
> not
> > practical since it reduces James scalability. In a very busy server
> where
> > there many (~ 100 users at any time at peak), it can make the heap
> exhausted
> > very fast.
>
> This is not possible because is not handled in james but in javamail.
> You are using getFlags that is a javamail method. Maybe in james 3.0 (a
> lot of months away) we'll try to remove javamail from james but you will
> not have the getFlags() method and other javamail specific operations.
>
> You can limit the maximum message size and the number of spool threads
> to limit the maximum memory usage.
>
> Maybe a different workaround for your problem does exists: why are you
> using getFlags()? what do you read/store in mimemessage flags? Can you
> use the Mail's attributes for the same purpose? Mail's attributes don't
> need javamail to be invoked (we don't need to even parse the
> mimemessage) and is more memory friendly.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
> Is it possible to store it into temporary file? Using more memory does not
> guarantee the stability, unless the email size is limited. But this is not
> practical since it reduces James scalability. In a very busy server where
> there many (~ 100 users at any time at peak), it can make the heap exhausted
> very fast.
This is not possible because is not handled in james but in javamail.
You are using getFlags that is a javamail method. Maybe in james 3.0 (a
lot of months away) we'll try to remove javamail from james but you will
not have the getFlags() method and other javamail specific operations.
You can limit the maximum message size and the number of spool threads
to limit the maximum memory usage.
Maybe a different workaround for your problem does exists: why are you
using getFlags()? what do you read/store in mimemessage flags? Can you
use the Mail's attributes for the same purpose? Mail's attributes don't
need javamail to be invoked (we don't need to even parse the
mimemessage) and is more memory friendly.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
Hi Stefano,
I have tried
ByteArrayInputStream headersIn = new
SharedByteArrayInputStream(headers.toByteArray());
but it still runs out of heap.
I used
-Xms512M -Xmx768M
but it made my system sluggish with swapping memory-file.
Is it possible to store it into temporary file? Using more memory does not
guarantee the stability, unless the email size is limited. But this is not
practical since it reduces James scalability. In a very busy server where
there many (~ 100 users at any time at peak), it can make the heap exhausted
very fast.
Thanks.
Regards,
Edward
On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> Edward Tan wrote:
> > Yes!
> >
> > } catch(Throwable t) {
> > System.out.println(t.getMessage());
> > t.printStackTrace();
> >
> > ====>
> >
> > Java heap space
> > java.lang.OutOfMemoryError: Java heap space
>
> The only solution is to increase the memory available to java (-Xmx500M
> for example)
>
> Otherwise you could try the com.sun.mail.util.SharedByteArrayInputStream
> way to see if javamail will require less memory. If you try this way and
> it works please tell us so we can decide wether to include it in our
> codebase or not.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
Edward Tan wrote:
> Yes!
>
> } catch(Throwable t) {
> System.out.println(t.getMessage());
> t.printStackTrace();
>
> ====>
>
> Java heap space
> java.lang.OutOfMemoryError: Java heap space
The only solution is to increase the memory available to java (-Xmx500M
for example)
Otherwise you could try the com.sun.mail.util.SharedByteArrayInputStream
way to see if javamail will require less memory. If you try this way and
it works please tell us so we can decide wether to include it in our
codebase or not.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
Yes!
} catch(Throwable t) {
System.out.println(t.getMessage());
t.printStackTrace();
====>
Java heap space
java.lang.OutOfMemoryError: Java heap space
On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> Edward Tan wrote:
> > The exception is caught somewhere, so I set at earlier calling function
> > stack.
> > But
> >
> > message is null
> >
> > and it goes to finally.
>
> If it execute the finally, the message is null and the 2
> System.out.printlm are not called then it could be a throwable (probably
> an error, an out of memory error?)
>
> If you can reproduce you could add a
> } catch (Throwable t) {
> // some output code
> throw t;
> } finally ...
>
> Stefano
>
> > Somewhere the exception is caught and never thrown back.
> >
> > ------------------
> > private synchronized void loadMessage() throws MessagingException {
> > if (message != null) {
> > //Another thread has already loaded this message
> > return;
> > }
> > InputStream in = null;
> > try {
> > in = source.getInputStream();
> > headers = new MailHeaders(in);
> >
> > ByteArrayInputStream headersIn
> > = new ByteArrayInputStream(headers.toByteArray());
> > in = new SequenceInputStream(headersIn, in);
> >
> > message = new MimeMessage(session, in);
> > } catch (IOException ioe) {
> > System.out.println(ioe);
> > throw new MessagingException("Unable to parse stream: " +
> > ioe.getMessage(), ioe);
> > } catch (Exception e) {
> > System.out.println(e);
> > e.printStackTrace();
> > } finally {
> > IOUtil.shutdownStream(in);
> > }
> > }
> > ----------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
Edward Tan wrote:
> The exception is caught somewhere, so I set at earlier calling function
> stack.
> But
>
> message is null
>
> and it goes to finally.
If it execute the finally, the message is null and the 2
System.out.printlm are not called then it could be a throwable (probably
an error, an out of memory error?)
If you can reproduce you could add a
} catch (Throwable t) {
// some output code
throw t;
} finally ...
Stefano
> Somewhere the exception is caught and never thrown back.
>
> ------------------
> private synchronized void loadMessage() throws MessagingException {
> if (message != null) {
> //Another thread has already loaded this message
> return;
> }
> InputStream in = null;
> try {
> in = source.getInputStream();
> headers = new MailHeaders(in);
>
> ByteArrayInputStream headersIn
> = new ByteArrayInputStream(headers.toByteArray());
> in = new SequenceInputStream(headersIn, in);
>
> message = new MimeMessage(session, in);
> } catch (IOException ioe) {
> System.out.println(ioe);
> throw new MessagingException("Unable to parse stream: " +
> ioe.getMessage(), ioe);
> } catch (Exception e) {
> System.out.println(e);
> e.printStackTrace();
> } finally {
> IOUtil.shutdownStream(in);
> }
> }
> ----------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
The exception is caught somewhere, so I set at earlier calling function
stack.
But
message is null
and it goes to finally.
Somewhere the exception is caught and never thrown back.
------------------
private synchronized void loadMessage() throws MessagingException {
if (message != null) {
//Another thread has already loaded this message
return;
}
InputStream in = null;
try {
in = source.getInputStream();
headers = new MailHeaders(in);
ByteArrayInputStream headersIn
= new ByteArrayInputStream(headers.toByteArray());
in = new SequenceInputStream(headersIn, in);
message = new MimeMessage(session, in);
} catch (IOException ioe) {
System.out.println(ioe);
throw new MessagingException("Unable to parse stream: " +
ioe.getMessage(), ioe);
} catch (Exception e) {
System.out.println(e);
e.printStackTrace();
} finally {
IOUtil.shutdownStream(in);
}
}
----------------------------
On 12/30/05, Edward Tan <ta...@gmail.com> wrote:
>
> I haven't pasted the exception message since I am confused why my Eclipse
> couldn't stop at the generic Exception and goes directly to finally. Or
> maybe my eyes are too tired.
>
> The exception is caught here, inside
> org.apache.james.core.MimeMessageWrapper
> ----------------------
> private synchronized void loadMessage() throws MessagingException {
> if (message != null) {
> //Another thread has already loaded this message
> return;
> }
> InputStream in = null;
> try {
> in = source.getInputStream();
> headers = new MailHeaders(in);
>
> ByteArrayInputStream headersIn
> = new ByteArrayInputStream(headers.toByteArray());
> in = new SequenceInputStream(headersIn, in);
>
> message = new MimeMessage(session, in);
> } catch (IOException ioe) {
> System.out.println(ioe);
> throw new MessagingException("Unable to parse stream: " +
> ioe.getMessage(), ioe);
> } catch (Exception e) {
> System.out.println(e);
> e.printStackTrace();
> } finally {
> IOUtil.shutdownStream(in);
> }
> }
> -------------------------
>
> But it doesn't stop at the breakpoints I set at
>
> throw new MessagingException....
>
> and
>
> e.printStackTrace....
>
> but it stops at IOUtil.shutdownStream(in);
>
> ----------------
>
> I add
> } catch (Exception e) {
> System.out.println(e);
> e.printStackTrace();
> in case it is other exception, but it was not caught here.
>
>
> -----------------
>
> I will try your suggestion.
>
>
> On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
> >
> > Edward Tan wrote:
> > > I just found out the culprit, it is not James code. It is
> > > javax.mail.MimeMessage
> > > [...]
> > > if (is instanceof SharedInputStream) {
> > > SharedInputStream sis = (SharedInputStream)is;
> > > contentStream = sis.newStream(sis.getPosition(), -1);
> > > } else {
> > > try {
> > > content = ASCIIUtility.getBytes(is);
> > > } catch (IOException ioex) {
> > > throw new MessagingException("IOException", ioex);
> > > }
> > > }
> >
> > I investigated a few months ago on using always
> > com.sun.mail.util.SharedByteArrayInputStream when loading message
> > sources to improve sun javamail performance.
> >
> > I replaced the following line in MimeMessageWrapper.loadMessage()
> >
> > ByteArrayInputStream headersIn = new
> > SharedByteArrayInputStream(headers.toByteArray());
> >
> > And the following in MimeMessageJDBCSource.getInputStream():
> >
> > InputStream in = new SharedByteArrayInputStream(headers);
> >
> > But I've had no time to do more tests.
> >
> > > -------------------------------------------
> > >
> > > I traced that the
> > >
> > > while ((len = is.read(buf, 0, size)) != -1)
> > > bos.write (buf, 0, len);
> > >
> > > causes the exception.
> > > -----------------------------------
> >
> > Again, What is the real exception? ;-)
> >
> > > With big attachment, this will crash. Any idea to avoid this?
> >
> > If you are confortable with james building try the above change and let
> > us know if it solve the problem!
> >
> > Stefano
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
> >
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
I haven't pasted the exception message since I am confused why my Eclipse
couldn't stop at the generic Exception and goes directly to finally. Or
maybe my eyes are too tired.
The exception is caught here, inside
org.apache.james.core.MimeMessageWrapper
----------------------
private synchronized void loadMessage() throws MessagingException {
if (message != null) {
//Another thread has already loaded this message
return;
}
InputStream in = null;
try {
in = source.getInputStream();
headers = new MailHeaders(in);
ByteArrayInputStream headersIn
= new ByteArrayInputStream(headers.toByteArray());
in = new SequenceInputStream(headersIn, in);
message = new MimeMessage(session, in);
} catch (IOException ioe) {
System.out.println(ioe);
throw new MessagingException("Unable to parse stream: " +
ioe.getMessage(), ioe);
} catch (Exception e) {
System.out.println(e);
e.printStackTrace();
} finally {
IOUtil.shutdownStream(in);
}
}
-------------------------
But it doesn't stop at the breakpoints I set at
throw new MessagingException....
and
e.printStackTrace....
but it stops at IOUtil.shutdownStream(in);
----------------
I add
} catch (Exception e) {
System.out.println(e);
e.printStackTrace();
in case it is other exception, but it was not caught here.
-----------------
I will try your suggestion.
On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> Edward Tan wrote:
> > I just found out the culprit, it is not James code. It is
> > javax.mail.MimeMessage
> > [...]
> > if (is instanceof SharedInputStream) {
> > SharedInputStream sis = (SharedInputStream)is;
> > contentStream = sis.newStream(sis.getPosition(), -1);
> > } else {
> > try {
> > content = ASCIIUtility.getBytes(is);
> > } catch (IOException ioex) {
> > throw new MessagingException("IOException", ioex);
> > }
> > }
>
> I investigated a few months ago on using always
> com.sun.mail.util.SharedByteArrayInputStream when loading message
> sources to improve sun javamail performance.
>
> I replaced the following line in MimeMessageWrapper.loadMessage()
>
> ByteArrayInputStream headersIn = new
> SharedByteArrayInputStream(headers.toByteArray());
>
> And the following in MimeMessageJDBCSource.getInputStream():
>
> InputStream in = new SharedByteArrayInputStream(headers);
>
> But I've had no time to do more tests.
>
> > -------------------------------------------
> >
> > I traced that the
> >
> > while ((len = is.read(buf, 0, size)) != -1)
> > bos.write(buf, 0, len);
> >
> > causes the exception.
> > -----------------------------------
>
> Again, What is the real exception? ;-)
>
> > With big attachment, this will crash. Any idea to avoid this?
>
> If you are confortable with james building try the above change and let
> us know if it solve the problem!
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
Edward Tan wrote:
> I just found out the culprit, it is not James code. It is
> javax.mail.MimeMessage
> [...]
> if (is instanceof SharedInputStream) {
> SharedInputStream sis = (SharedInputStream)is;
> contentStream = sis.newStream(sis.getPosition(), -1);
> } else {
> try {
> content = ASCIIUtility.getBytes(is);
> } catch (IOException ioex) {
> throw new MessagingException("IOException", ioex);
> }
> }
I investigated a few months ago on using always
com.sun.mail.util.SharedByteArrayInputStream when loading message
sources to improve sun javamail performance.
I replaced the following line in MimeMessageWrapper.loadMessage()
ByteArrayInputStream headersIn = new
SharedByteArrayInputStream(headers.toByteArray());
And the following in MimeMessageJDBCSource.getInputStream():
InputStream in = new SharedByteArrayInputStream(headers);
But I've had no time to do more tests.
> -------------------------------------------
>
> I traced that the
>
> while ((len = is.read(buf, 0, size)) != -1)
> bos.write(buf, 0, len);
>
> causes the exception.
> -----------------------------------
Again, What is the real exception? ;-)
> With big attachment, this will crash. Any idea to avoid this?
If you are confortable with james building try the above change and let
us know if it solve the problem!
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
I just found out the culprit, it is not James code. It is
javax.mail.MimeMessage
Exactly:
/**
* Byte array that holds the bytes of this Message's content.
*/
protected byte[] content;
.......
content = ASCIIUtility.getBytes(is);
in:
protected void parse(InputStream is) throws MessagingException {
if (!(is instanceof ByteArrayInputStream) &&
!(is instanceof BufferedInputStream) &&
!(is instanceof SharedInputStream))
is = new BufferedInputStream(is);
headers = createInternetHeaders(is);
if (is instanceof SharedInputStream) {
SharedInputStream sis = (SharedInputStream)is;
contentStream = sis.newStream(sis.getPosition(), -1);
} else {
try {
content = ASCIIUtility.getBytes(is);
} catch (IOException ioex) {
throw new MessagingException("IOException", ioex);
}
}
---------------------------------------------------------------
The ASCIIUtility.getBytes(s):
public static byte[] getBytes(InputStream is) throws IOException {
int len;
int size = 1024;
byte [] buf;
if (is instanceof ByteArrayInputStream) {
size = is.available();
buf = new byte[size];
len = is.read(buf, 0, size);
}
else {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
buf = new byte[size];
while ((len = is.read(buf, 0, size)) != -1)
bos.write(buf, 0, len);
buf = bos.toByteArray();
}
return buf;
}
}
-------------------------------------------
I traced that the
while ((len = is.read(buf, 0, size)) != -1)
bos.write(buf, 0, len);
causes the exception.
-----------------------------------
With big attachment, this will crash. Any idea to avoid this?
On 12/30/05, Edward Tan <ta...@gmail.com> wrote:
>
> Hi,
>
> James suddenly crash when I receive > 30MB attachment. I didn't check the
> threshold of message size that causes this. I just happened to send 34 MB
> attachment through Thunderbird to James. The problem is caused by my mailet
> which tries to get the flag:
>
>
> in my my mailet:
>
> message = mail.getMessage();
>
> .....
>
>
> // FLAGS
> ---> (exception when calling this:) Flags flags =
> message.getFlags();
>
>
> The exception is caught in org.apache.james.transport.JamesSpoolManager
>
> line 233:
> } catch (Throwable e) {
> // This is a strange error situation that shouldn't
> ordinarily
> // happen
> StringBuffer exceptionBuffer =
> new StringBuffer(64)
> .append("Exception in processor <")
> .append(processorName)
> .append(">");
> getLogger().error(exceptionBuffer.toString(), e);
> if (processorName.equals(Mail.ERROR)) {
> // We got an error on the error processor...
> // kill the message
> mail.setState(Mail.GHOST);
> mail.setErrorMessage(e.getMessage());
>
>
> Has anyone experienced before?
>
> Any limit on message size?
>
> Regards,
> Edward
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Edward Tan <ta...@gmail.com>.
I wrapped it with MimeMessageWrapper and it still crashed.
The culprit is inside javax.mail
public static byte[] getBytes(InputStream is) throws IOException {
int len;
int size = 1024;
byte [] buf;
if (is instanceof ByteArrayInputStream) {
size = is.available();
buf = new byte[size];
len = is.read(buf, 0, size);
}
else {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
buf = new byte[size];
while ((len = is.read(buf, 0, size)) != -1)
bos.write(buf, 0, len);
buf = bos.toByteArray();
}
return buf;
}
-------------------
The
while ((len = is.read(buf, 0, size)) != -1)
bos.write(buf, 0, len);
mysteriously throw exception on bos.write(buf, 0, len);
in the middle of the big file size.
Anything I should do with the java VM args?
I used 1GB RAM, AMD 64, Fedore x86_64 and Sun Jdk 1.5_06
On 12/30/05, Stefano Bagnara <ap...@bago.org> wrote:
>
> Where is the stack trace?
> We need at least to know the Exception type and the error message.
> The logfile would also help.
>
> Maybe the problem is that message.getFlags() was not wrapped in the
> MimeMessageWrapper for 2.2.0. I committed yesterday a patch to
> MimeMessageWrapper adding a few missing wrapping methods.
>
> Stefano
>
> Edward Tan wrote:
> > Hi,
> >
> > James suddenly crash when I receive > 30MB attachment. I didn't check
> the
> > threshold of message size that causes this. I just happened to send 34
> MB
> > attachment through Thunderbird to James. The problem is caused by my
> mailet
> > which tries to get the flag:
> >
> >
> > in my my mailet:
> >
> > message = mail.getMessage();
> >
> > .....
> >
> >
> > // FLAGS
> > ---> (exception when calling this:) Flags flags =
> > message.getFlags();
> >
> >
> > The exception is caught in org.apache.james.transport.JamesSpoolManager
> >
> > line 233:
> > } catch (Throwable e) {
> > // This is a strange error situation that shouldn't
> > ordinarily
> > // happen
> > StringBuffer exceptionBuffer =
> > new StringBuffer(64)
> > .append("Exception in processor <")
> > .append(processorName)
> > .append(">");
> > getLogger().error(exceptionBuffer.toString(), e);
> > if (processorName.equals(Mail.ERROR)) {
> > // We got an error on the error processor...
> > // kill the message
> > mail.setState(Mail.GHOST);
> > mail.setErrorMessage(e.getMessage());
> >
> >
> > Has anyone experienced before?
> >
> > Any limit on message size?
> >
> > Regards,
> > Edward
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
Re: Fatal Exception when receiving email > 30 MB
Posted by Stefano Bagnara <ap...@bago.org>.
Where is the stack trace?
We need at least to know the Exception type and the error message.
The logfile would also help.
Maybe the problem is that message.getFlags() was not wrapped in the
MimeMessageWrapper for 2.2.0. I committed yesterday a patch to
MimeMessageWrapper adding a few missing wrapping methods.
Stefano
Edward Tan wrote:
> Hi,
>
> James suddenly crash when I receive > 30MB attachment. I didn't check the
> threshold of message size that causes this. I just happened to send 34 MB
> attachment through Thunderbird to James. The problem is caused by my mailet
> which tries to get the flag:
>
>
> in my my mailet:
>
> message = mail.getMessage();
>
> .....
>
>
> // FLAGS
> ---> (exception when calling this:) Flags flags =
> message.getFlags();
>
>
> The exception is caught in org.apache.james.transport.JamesSpoolManager
>
> line 233:
> } catch (Throwable e) {
> // This is a strange error situation that shouldn't
> ordinarily
> // happen
> StringBuffer exceptionBuffer =
> new StringBuffer(64)
> .append("Exception in processor <")
> .append(processorName)
> .append(">");
> getLogger().error(exceptionBuffer.toString(), e);
> if (processorName.equals(Mail.ERROR)) {
> // We got an error on the error processor...
> // kill the message
> mail.setState(Mail.GHOST);
> mail.setErrorMessage(e.getMessage());
>
>
> Has anyone experienced before?
>
> Any limit on message size?
>
> Regards,
> Edward
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org